[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:07:49 dist-test-slave-d2pk krb5kdc[18768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:07:49 dist-test-slave-d2pk krb5kdc[18768](info): set up 2 sockets
Oct 25 20:07:49 dist-test-slave-d2pk krb5kdc[18768](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251025 20:07:51.421890 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:07:51 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:07:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:51.541792 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:32777
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:51.643293 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:51.643496 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:51.643545 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:51.646517 18784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:07:51.646570 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:51.646592 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:51.646610 18784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:07:51.646626 18784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:07:51.650388 18784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:51.651295 18784 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:51.652073 18784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:51.657966 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:51.658109 18784 server_base.cc:1047] running on GCE node
W20251025 20:07:51.657963 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:51.657966 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:51.658762 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:51.659565 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:51.660713 18784 hybrid_clock.cc:648] HybridClock initialized: now 1761422871660705 us; error 24 us; skew 500 ppm
Oct 25 20:07:51 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:51.663293 18784 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:07:51.664305 18784 webserver.cc:492] Webserver started at http://127.18.82.126:34287/ using document root <none> and password file <none>
I20251025 20:07:51.664847 18784 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:51.664902 18784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:51.665084 18784 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:51.666803 18784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "7ca51de9b2974db083dc5710b3d1102b"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "ba221319e4456d3f9b8918e6f1211d61"
server_key_iv: "acbc5f357c12eb4dac6af96899a0cd5d"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.667294 18784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "7ca51de9b2974db083dc5710b3d1102b"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "ba221319e4456d3f9b8918e6f1211d61"
server_key_iv: "acbc5f357c12eb4dac6af96899a0cd5d"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.670495 18784 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:07:51.672641 18799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:51.673571 18784 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:07:51.673702 18784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "7ca51de9b2974db083dc5710b3d1102b"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "ba221319e4456d3f9b8918e6f1211d61"
server_key_iv: "acbc5f357c12eb4dac6af96899a0cd5d"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.673789 18784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:51.696980 18784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:51.699611 18784 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:51.699764 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:51.706284 18851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:32777 every 8 connection(s)
I20251025 20:07:51.706296 18784 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:32777
I20251025 20:07:51.707340 18784 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:07:51.710079 18852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:51.715101 18852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: Bootstrap starting.
I20251025 20:07:51.717149 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 18784
I20251025 20:07:51.717281 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:07:51.717720 18852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:51.718478 18761 external_mini_cluster.cc:1468] Setting key 90083933ce6f4715b1a332ccdb0b374b
I20251025 20:07:51.718536 18852 log.cc:826] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:51.720507 18852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: No bootstrap required, opened a new log
I20251025 20:07:51.723111 18852 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } }
I20251025 20:07:51.723279 18852 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:51.723335 18852 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca51de9b2974db083dc5710b3d1102b, State: Initialized, Role: FOLLOWER
I20251025 20:07:51.723722 18852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } }
I20251025 20:07:51.723840 18852 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:07:51.723902 18852 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:07:51.723985 18852 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:51.724717 18852 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } }
I20251025 20:07:51.725021 18852 leader_election.cc:304] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ca51de9b2974db083dc5710b3d1102b; no voters:
I20251025 20:07:51.725252 18852 leader_election.cc:290] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:07:51.725347 18857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:51.725557 18857 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [term 1 LEADER]: Becoming Leader. State: Replica: 7ca51de9b2974db083dc5710b3d1102b, State: Running, Role: LEADER
I20251025 20:07:51.725831 18857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } }
I20251025 20:07:51.726442 18852 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:07:51.727614 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca51de9b2974db083dc5710b3d1102b. Latest consensus state: current_term: 1 leader_uuid: "7ca51de9b2974db083dc5710b3d1102b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } } }
I20251025 20:07:51.727674 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ca51de9b2974db083dc5710b3d1102b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca51de9b2974db083dc5710b3d1102b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 32777 } } }
I20251025 20:07:51.727747 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [sys.catalog]: This master's current role is: LEADER
I20251025 20:07:51.727795 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b [sys.catalog]: This master's current role is: LEADER
Oct 25 20:07:51 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
W20251025 20:07:51.731372 18872 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:07:51.731436 18872 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:07:51.731506 18866 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:07:51.732303 18866 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:07:51.736536 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:51.720774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.18.82.126:32777)
1025 20:07:51.721194 (+ 420us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:51.721205 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:51.721756 (+ 551us) server_negotiation.cc:408] Connection header received
1025 20:07:51.723528 (+ 1772us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:51.723540 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:51.723783 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:51.724063 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:51.725767 (+ 1704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:51.726731 (+ 964us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:51.727649 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:51.727981 (+ 332us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:51.731979 (+ 3998us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:51.731994 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:51.732002 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:51.732030 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:51.733663 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:51.734081 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:51.734086 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:51.734089 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:51.734150 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:51.734434 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:51.734438 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:51.734439 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:51.734807 (+ 368us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:51.734993 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:51.735644 (+ 651us) server_negotiation.cc:300] Negotiation successful
1025 20:07:51.735907 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":135,"threads_started":1}
I20251025 20:07:51.738606 18866 catalog_manager.cc:1357] Generated new cluster ID: 3f6aa77a76ff47a69bd167a325a144eb
I20251025 20:07:51.738664 18866 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:07:51.768949 18866 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:07:51.770144 18866 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:07:51.780249 18866 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7ca51de9b2974db083dc5710b3d1102b: Generated new TSK 0
I20251025 20:07:51.781141 18866 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:51.832662 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:32777
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:51.940907 18880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:51.941087 18880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:51.941159 18880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:51.943934 18880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:51.943979 18880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:51.944044 18880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:07:51.947863 18880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:32777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:51.948745 18880 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:51.949457 18880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:51.955323 18888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:51.955317 18885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:51.955317 18886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:51.955485 18880 server_base.cc:1047] running on GCE node
I20251025 20:07:51.955889 18880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:51.956358 18880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:51.957562 18880 hybrid_clock.cc:648] HybridClock initialized: now 1761422871957555 us; error 25 us; skew 500 ppm
Oct 25 20:07:51 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:51.959894 18880 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:07:51.960847 18880 webserver.cc:492] Webserver started at http://127.18.82.65:45613/ using document root <none> and password file <none>
I20251025 20:07:51.961319 18880 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:51.961369 18880 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:51.961585 18880 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:51.963073 18880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "20034b41b073480db9ac4e90f3cd1a07"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "c8bf4bf5d1bdc08b0a546d4a6d45c983"
server_key_iv: "c20f0f3626ecde003863d3086e7f2f6c"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.963521 18880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "20034b41b073480db9ac4e90f3cd1a07"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "c8bf4bf5d1bdc08b0a546d4a6d45c983"
server_key_iv: "c20f0f3626ecde003863d3086e7f2f6c"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.966524 18880 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:07:51.968624 18895 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:51.969497 18880 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:51.969626 18880 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "20034b41b073480db9ac4e90f3cd1a07"
format_stamp: "Formatted at 2025-10-25 20:07:51 on dist-test-slave-d2pk"
server_key: "c8bf4bf5d1bdc08b0a546d4a6d45c983"
server_key_iv: "c20f0f3626ecde003863d3086e7f2f6c"
server_key_version: "encryptionkey@0"
I20251025 20:07:51.969703 18880 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:51.983357 18880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:51.985960 18880 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:51.986104 18880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:51.986624 18880 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:51.987468 18880 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:51.987517 18880 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:51.987574 18880 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:51.987602 18880 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:51.995652 18880 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:39923
I20251025 20:07:51.995688 19008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:39923 every 8 connection(s)
I20251025 20:07:51.996454 18880 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:07:51.998010 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 18880
I20251025 20:07:51.998102 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:07:51.998306 18761 external_mini_cluster.cc:1468] Setting key e29561dffb97eaa1207e4760476fe3a9
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
I20251025 20:07:52.008932 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:51.998146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:43877 (local address 127.18.82.126:32777)
1025 20:07:51.998311 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:51.998317 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:51.998986 (+ 669us) server_negotiation.cc:408] Connection header received
1025 20:07:51.999902 (+ 916us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:51.999906 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:51.999949 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:52.000011 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:52.002000 (+ 1989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.002475 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.003259 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.003401 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.005786 (+ 2385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:52.005802 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:52.005804 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:52.005826 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:52.007246 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.007823 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.007826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.007827 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.007868 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.008119 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.008121 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.008122 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.008245 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:52.008308 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:52.008686 (+ 378us) server_negotiation.cc:300] Negotiation successful
1025 20:07:52.008814 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 20:07:52.009497 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:51.998433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:32777 (local address 127.18.82.65:43877)
1025 20:07:51.998839 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:51.998880 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:51.999702 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:52.000163 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:52.000175 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:52.000727 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:52.001866 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.001880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.002594 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.002600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:52.003146 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.003159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.003361 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.003966 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:52.004004 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:52.005645 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:52.007372 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.007378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.007388 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.007717 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.007953 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.007959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.007963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.008052 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.008310 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:52.008315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:52.008507 (+ 192us) client_negotiation.cc:769] Sending connection context
1025 20:07:52.008673 (+ 166us) client_negotiation.cc:240] Negotiation successful
1025 20:07:52.008878 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
I20251025 20:07:52.010663 19009 heartbeater.cc:344] Connected to a master server at 127.18.82.126:32777
I20251025 20:07:52.010892 19009 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:52.011377 19009 heartbeater.cc:507] Master 127.18.82.126:32777 requested a full tablet report, sending...
I20251025 20:07:52.012775 18816 ts_manager.cc:194] Registered new tserver with Master: 20034b41b073480db9ac4e90f3cd1a07 (127.18.82.65:39923)
I20251025 20:07:52.013972 18816 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:43877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:52.041710 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:32777
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:52.144536 19016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:52.144726 19016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:52.144778 19016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:52.147859 19016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:52.147915 19016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:52.147990 19016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:07:52.151888 19016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:32777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:52.152788 19016 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:52.153558 19016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:52.160149 19022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:52.160248 19016 server_base.cc:1047] running on GCE node
W20251025 20:07:52.160202 19024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:52.160151 19021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:52.160660 19016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:52.161118 19016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:52.162249 19016 hybrid_clock.cc:648] HybridClock initialized: now 1761422872162241 us; error 23 us; skew 500 ppm
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:52.164592 19016 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:07:52.165618 19016 webserver.cc:492] Webserver started at http://127.18.82.66:41035/ using document root <none> and password file <none>
I20251025 20:07:52.166121 19016 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:52.166174 19016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:52.166383 19016 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:52.167999 19016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "f51a9a02ed9445568161caa8baf524f3"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "02cc2fa67fadb26d26cd690c689e9089"
server_key_iv: "d78e332652280716b55dba543165c15f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.168465 19016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "f51a9a02ed9445568161caa8baf524f3"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "02cc2fa67fadb26d26cd690c689e9089"
server_key_iv: "d78e332652280716b55dba543165c15f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.171561 19016 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 20:07:52.173470 19031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.174484 19016 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:52.174578 19016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "f51a9a02ed9445568161caa8baf524f3"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "02cc2fa67fadb26d26cd690c689e9089"
server_key_iv: "d78e332652280716b55dba543165c15f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.174667 19016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:52.188024 19016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:52.190681 19016 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:52.190850 19016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:52.191347 19016 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:52.192152 19016 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:52.192195 19016 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.192234 19016 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:52.192257 19016 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.200538 19016 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:37711
I20251025 20:07:52.200575 19144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:37711 every 8 connection(s)
I20251025 20:07:52.201391 19016 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:07:52.206092 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19016
I20251025 20:07:52.206180 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:07:52.206451 18761 external_mini_cluster.cc:1468] Setting key 28e6058c558798470ce7432642b4baa3
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:52.213359 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.202997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:43655 (local address 127.18.82.126:32777)
1025 20:07:52.203138 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:52.203142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:52.203770 (+ 628us) server_negotiation.cc:408] Connection header received
1025 20:07:52.204558 (+ 788us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:52.204562 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:52.204624 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:52.204699 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:52.206243 (+ 1544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.206973 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.207725 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.207890 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.210091 (+ 2201us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:52.210108 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:52.210111 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:52.210138 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:52.211653 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.212113 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.212116 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.212118 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.212159 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.212413 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.212419 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.212420 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.212558 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:52.212646 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:52.213085 (+ 439us) server_negotiation.cc:300] Negotiation successful
1025 20:07:52.213203 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 20:07:52.213989 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.203225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:32777 (local address 127.18.82.66:43655)
1025 20:07:52.203631 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:52.203672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:52.204371 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:52.204855 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:52.204863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:52.205205 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:52.206087 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.206101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.207094 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.207097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:52.207605 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.207616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.207859 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.208427 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:52.208449 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:52.209955 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:52.211766 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.211771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.211782 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.212002 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.212240 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.212243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.212245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.212323 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.212663 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:52.212670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:52.212869 (+ 199us) client_negotiation.cc:769] Sending connection context
1025 20:07:52.213039 (+ 170us) client_negotiation.cc:240] Negotiation successful
1025 20:07:52.213284 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:07:52.215159 19145 heartbeater.cc:344] Connected to a master server at 127.18.82.126:32777
I20251025 20:07:52.215373 19145 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:52.215842 19145 heartbeater.cc:507] Master 127.18.82.126:32777 requested a full tablet report, sending...
I20251025 20:07:52.216827 18816 ts_manager.cc:194] Registered new tserver with Master: f51a9a02ed9445568161caa8baf524f3 (127.18.82.66:37711)
I20251025 20:07:52.217391 18816 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:43655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:52.249562 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:32777
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:52.349041 19152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:52.349268 19152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:52.349365 19152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:52.352312 19152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:52.352385 19152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:52.352492 19152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:07:52.356089 19152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:32777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:52.357110 19152 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:52.357975 19152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:52.364113 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:52.364114 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:52.364207 19152 server_base.cc:1047] running on GCE node
W20251025 20:07:52.364104 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:52.364661 19152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:52.365162 19152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:52.366312 19152 hybrid_clock.cc:648] HybridClock initialized: now 1761422872366304 us; error 32 us; skew 500 ppm
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:52.368749 19152 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:07:52.369715 19152 webserver.cc:492] Webserver started at http://127.18.82.67:43017/ using document root <none> and password file <none>
I20251025 20:07:52.370205 19152 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:52.370254 19152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:52.370426 19152 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:52.371919 19152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "2ea4fa589c394c0ab552d90643fb36b0"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "797dcc9f4fecd4bbc2c631fa0f522cad"
server_key_iv: "9aadd14fb00c0674852856d72500349f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.372359 19152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "2ea4fa589c394c0ab552d90643fb36b0"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "797dcc9f4fecd4bbc2c631fa0f522cad"
server_key_iv: "9aadd14fb00c0674852856d72500349f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.375535 19152 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251025 20:07:52.377534 19167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.378554 19152 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:07:52.378651 19152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "2ea4fa589c394c0ab552d90643fb36b0"
format_stamp: "Formatted at 2025-10-25 20:07:52 on dist-test-slave-d2pk"
server_key: "797dcc9f4fecd4bbc2c631fa0f522cad"
server_key_iv: "9aadd14fb00c0674852856d72500349f"
server_key_version: "encryptionkey@0"
I20251025 20:07:52.378732 19152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:52.396456 19152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:52.399055 19152 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:52.399205 19152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:52.399690 19152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:52.400460 19152 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:52.400503 19152 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.400558 19152 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:52.400583 19152 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:52.408576 19152 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:43111
I20251025 20:07:52.408610 19280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:43111 every 8 connection(s)
I20251025 20:07:52.409466 19152 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:07:52.414611 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19152
I20251025 20:07:52.414717 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:07:52.414932 18761 external_mini_cluster.cc:1468] Setting key 5357e6b565c6fe91e8ec1bd025780687
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:52.420555 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.411045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:42351 (local address 127.18.82.126:32777)
1025 20:07:52.411163 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:52.411166 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:52.411776 (+ 610us) server_negotiation.cc:408] Connection header received
1025 20:07:52.412502 (+ 726us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:52.412506 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:52.412558 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:52.412628 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:52.414239 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.414904 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.415546 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.415715 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.417897 (+ 2182us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:52.417923 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:52.417926 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:52.417950 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:52.419153 (+ 1203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.419565 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.419568 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.419569 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.419615 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.419797 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.419799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.419800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.419923 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:52.419987 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:52.420355 (+ 368us) server_negotiation.cc:300] Negotiation successful
1025 20:07:52.420447 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:07:52.421177 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.411229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:32777 (local address 127.18.82.67:42351)
1025 20:07:52.411630 (+ 401us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:52.411663 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:52.412328 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:52.412866 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:52.412875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:52.413211 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:52.414111 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.414126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.415022 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.415026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:52.415431 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:52.415443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.415611 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.416216 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:52.416247 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:52.417763 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:52.419258 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.419262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.419271 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.419488 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.419670 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:52.419671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:52.419673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:52.419741 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:52.419985 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:52.419989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:52.420182 (+ 193us) client_negotiation.cc:769] Sending connection context
1025 20:07:52.420332 (+ 150us) client_negotiation.cc:240] Negotiation successful
1025 20:07:52.420518 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20251025 20:07:52.422152 19281 heartbeater.cc:344] Connected to a master server at 127.18.82.126:32777
I20251025 20:07:52.422358 19281 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:52.422822 19281 heartbeater.cc:507] Master 127.18.82.126:32777 requested a full tablet report, sending...
I20251025 20:07:52.423717 18816 ts_manager.cc:194] Registered new tserver with Master: 2ea4fa589c394c0ab552d90643fb36b0 (127.18.82.67:43111)
I20251025 20:07:52.424228 18816 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:42351
I20251025 20:07:52.428941 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:52.451081 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.443427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.18.82.65:39923)
1025 20:07:52.443770 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:52.443776 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:52.443799 (+ 23us) server_negotiation.cc:408] Connection header received
1025 20:07:52.443854 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:52.443858 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:52.444028 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:52.444140 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:52.444789 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.445418 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.445960 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.446128 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.448126 (+ 1998us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:52.448148 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:52.448159 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:52.448182 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:52.449842 (+ 1660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.450162 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.450167 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.450168 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.450210 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.450348 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.450350 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.450351 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.450503 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:52.450657 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:52.450759 (+ 102us) server_negotiation.cc:300] Negotiation successful
1025 20:07:52.450896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":138,"threads_started":1}
W20251025 20:07:52.451989 18923 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46240
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[18768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:52.473242 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:52.466916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.18.82.65:39923)
1025 20:07:52.467062 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:52.467066 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:52.467144 (+ 78us) server_negotiation.cc:408] Connection header received
1025 20:07:52.467267 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:52.467270 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:52.467317 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:52.467380 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:52.468065 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.468516 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:52.469044 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:52.469161 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:52.471063 (+ 1902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:52.471080 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:52.471083 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:52.471106 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:52.472224 (+ 1118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.472535 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.472537 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.472538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.472568 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:52.472718 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:52.472720 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:52.472721 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:52.472824 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:52.472916 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:52.473054 (+ 138us) server_negotiation.cc:300] Negotiation successful
1025 20:07:52.473132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 20:07:52.474145 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 18880
I20251025 20:07:52.480392 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19016
I20251025 20:07:52.488247 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19152
I20251025 20:07:52.493992 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 18784
2025-10-25T20:07:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2929 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[19295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[19295](info): set up 2 sockets
Oct 25 20:07:52 dist-test-slave-d2pk krb5kdc[19295](info): commencing operation
krb5kdc: starting...
W20251025 20:07:54.321867 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.806s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:07:54 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:07:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:54.444468 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:36739
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:54.543107 19311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:54.543305 19311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:54.543401 19311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:54.546470 19311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:07:54.546521 19311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:54.546558 19311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:54.546600 19311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:07:54.546636 19311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:07:54.550469 19311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:36739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:54.551376 19311 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:54.552157 19311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:54.557466 19317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:54.557487 19316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:54.557607 19311 server_base.cc:1047] running on GCE node
W20251025 20:07:54.557529 19319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:54.558223 19311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:54.559057 19311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:54.560199 19311 hybrid_clock.cc:648] HybridClock initialized: now 1761422874560189 us; error 22 us; skew 500 ppm
Oct 25 20:07:54 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:54.562726 19311 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:07:54.563655 19311 webserver.cc:492] Webserver started at http://127.18.82.126:36273/ using document root <none> and password file <none>
I20251025 20:07:54.564146 19311 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:54.564196 19311 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:54.564368 19311 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:54.566120 19311 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "c7a32fb089a348439afb643a8d5cad76"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "cb0cbee4cb0c3b71f6de051237c4954d"
server_key_iv: "ea7e7408bcef68d2984830a98470ab48"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.566613 19311 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "c7a32fb089a348439afb643a8d5cad76"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "cb0cbee4cb0c3b71f6de051237c4954d"
server_key_iv: "ea7e7408bcef68d2984830a98470ab48"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.569758 19311 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:07:54.571780 19326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:54.572681 19311 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:54.572781 19311 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "c7a32fb089a348439afb643a8d5cad76"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "cb0cbee4cb0c3b71f6de051237c4954d"
server_key_iv: "ea7e7408bcef68d2984830a98470ab48"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.572860 19311 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:54.588654 19311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:54.591022 19311 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:54.591172 19311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:54.597795 19311 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:36739
I20251025 20:07:54.597810 19378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:36739 every 8 connection(s)
I20251025 20:07:54.598723 19311 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:07:54.601181 19379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:54.606305 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76: Bootstrap starting.
I20251025 20:07:54.608812 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:54.608953 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19311
I20251025 20:07:54.609058 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:07:54.609302 18761 external_mini_cluster.cc:1468] Setting key e12694cee126115bdcf42f381deebf67
I20251025 20:07:54.609578 19379 log.cc:826] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:54.611585 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76: No bootstrap required, opened a new log
I20251025 20:07:54.614344 19379 raft_consensus.cc:359] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } }
I20251025 20:07:54.614543 19379 raft_consensus.cc:385] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:54.614631 19379 raft_consensus.cc:740] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a32fb089a348439afb643a8d5cad76, State: Initialized, Role: FOLLOWER
I20251025 20:07:54.615051 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } }
I20251025 20:07:54.615175 19379 raft_consensus.cc:399] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:07:54.615232 19379 raft_consensus.cc:493] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:07:54.615310 19379 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:54.616314 19379 raft_consensus.cc:515] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } }
I20251025 20:07:54.616709 19379 leader_election.cc:304] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a32fb089a348439afb643a8d5cad76; no voters:
I20251025 20:07:54.617039 19379 leader_election.cc:290] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:07:54.617322 19385 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 1 FOLLOWER]: Leader election won for term 1
Oct 25 20:07:54 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:54.617498 19385 raft_consensus.cc:697] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [term 1 LEADER]: Becoming Leader. State: Replica: c7a32fb089a348439afb643a8d5cad76, State: Running, Role: LEADER
I20251025 20:07:54.618191 19379 sys_catalog.cc:565] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:07:54.618306 19385 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } }
I20251025 20:07:54.619848 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7a32fb089a348439afb643a8d5cad76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } } }
I20251025 20:07:54.619951 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [sys.catalog]: This master's current role is: LEADER
I20251025 20:07:54.620182 19384 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7a32fb089a348439afb643a8d5cad76. Latest consensus state: current_term: 1 leader_uuid: "c7a32fb089a348439afb643a8d5cad76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a32fb089a348439afb643a8d5cad76" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36739 } } }
I20251025 20:07:54.620255 19384 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76 [sys.catalog]: This master's current role is: LEADER
I20251025 20:07:54.620282 19398 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:07:54.624143 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:54.610629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.18.82.126:36739)
1025 20:07:54.611065 (+ 436us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:54.611075 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:54.611110 (+ 35us) server_negotiation.cc:408] Connection header received
1025 20:07:54.611723 (+ 613us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:54.611745 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:54.612099 (+ 354us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:54.612438 (+ 339us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:54.613387 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.614492 (+ 1105us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:54.615512 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.615748 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:54.618084 (+ 2336us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:54.618101 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:54.618124 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:54.618154 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:54.621433 (+ 3279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:54.621922 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:54.621927 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:54.621932 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:54.622015 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:54.622327 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:54.622330 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:54.622332 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:54.622735 (+ 403us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:54.622928 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:54.623187 (+ 259us) server_negotiation.cc:300] Negotiation successful
1025 20:07:54.623461 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":115,"threads_started":1}
I20251025 20:07:54.624379 19398 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:07:54.628978 19398 catalog_manager.cc:1357] Generated new cluster ID: ea22d41f604a4304a10249bf32e03156
I20251025 20:07:54.629035 19398 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:07:54.647750 19398 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:07:54.648702 19398 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:07:54.653975 19398 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c7a32fb089a348439afb643a8d5cad76: Generated new TSK 0
I20251025 20:07:54.654575 19398 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:54.696491 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36739
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:54.795807 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:54.796042 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:54.796132 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:54.799253 19407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:54.799309 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:54.799429 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:07:54.803035 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:54.803942 19407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:54.804816 19407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:54.811048 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:54.811039 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:54.811039 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:54.811260 19407 server_base.cc:1047] running on GCE node
I20251025 20:07:54.811625 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:54.812108 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:54.813251 19407 hybrid_clock.cc:648] HybridClock initialized: now 1761422874813237 us; error 27 us; skew 500 ppm
Oct 25 20:07:54 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:54.815680 19407 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:07:54.816558 19407 webserver.cc:492] Webserver started at http://127.18.82.65:38563/ using document root <none> and password file <none>
I20251025 20:07:54.817075 19407 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:54.817126 19407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:54.817297 19407 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:54.818943 19407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "4a917a5b676740a6a52300f1e9c678d9"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "72d1ea29482f45b5616b55b295810bc0"
server_key_iv: "7f16180b34606fe3e81a544aa3f29e09"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.819397 19407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "4a917a5b676740a6a52300f1e9c678d9"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "72d1ea29482f45b5616b55b295810bc0"
server_key_iv: "7f16180b34606fe3e81a544aa3f29e09"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.822539 19407 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:07:54.824643 19422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:54.825528 19407 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:54.825631 19407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "4a917a5b676740a6a52300f1e9c678d9"
format_stamp: "Formatted at 2025-10-25 20:07:54 on dist-test-slave-d2pk"
server_key: "72d1ea29482f45b5616b55b295810bc0"
server_key_iv: "7f16180b34606fe3e81a544aa3f29e09"
server_key_version: "encryptionkey@0"
I20251025 20:07:54.825716 19407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:54.846773 19407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:54.853319 19407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:54.853487 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:54.854046 19407 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:54.854882 19407 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:54.854928 19407 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:54.854984 19407 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:54.855008 19407 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:54.863277 19407 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:37957
I20251025 20:07:54.863308 19535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:37957 every 8 connection(s)
I20251025 20:07:54.864113 19407 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:07:54 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:54.872054 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19407
I20251025 20:07:54.872160 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:07:54.872373 18761 external_mini_cluster.cc:1468] Setting key 58fbc00362056f9f4b417f98bfab21ea
I20251025 20:07:54.875131 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:54.865655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:55573 (local address 127.18.82.126:36739)
1025 20:07:54.865803 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:54.865808 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:54.866500 (+ 692us) server_negotiation.cc:408] Connection header received
1025 20:07:54.867276 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:54.867279 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:54.867327 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:54.867404 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:54.868937 (+ 1533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.869422 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:54.870016 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.870136 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:54.871892 (+ 1756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:54.871909 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:54.871912 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:54.871934 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:54.873270 (+ 1336us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:54.873777 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:54.873781 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:54.873783 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:54.873828 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:54.874136 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:54.874139 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:54.874141 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:54.874281 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:54.874370 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:54.874869 (+ 499us) server_negotiation.cc:300] Negotiation successful
1025 20:07:54.874983 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:07:54.875800 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:54.865936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36739 (local address 127.18.82.65:55573)
1025 20:07:54.866348 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:54.866383 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:54.867106 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:54.867548 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:54.867556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:54.867912 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:54.868824 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:54.868837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.869530 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:54.869533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:54.869918 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:54.869923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:54.870081 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:54.870519 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:54.870538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:54.871769 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:54.873358 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:54.873364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:54.873374 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:54.873676 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:54.873924 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:54.873927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:54.873930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:54.874053 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:54.874379 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:54.874386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:54.874624 (+ 238us) client_negotiation.cc:769] Sending connection context
1025 20:07:54.874818 (+ 194us) client_negotiation.cc:240] Negotiation successful
1025 20:07:54.875090 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20251025 20:07:54.876842 19536 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36739
I20251025 20:07:54.877122 19536 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:54.877789 19536 heartbeater.cc:507] Master 127.18.82.126:36739 requested a full tablet report, sending...
I20251025 20:07:54.879396 19343 ts_manager.cc:194] Registered new tserver with Master: 4a917a5b676740a6a52300f1e9c678d9 (127.18.82.65:37957)
I20251025 20:07:54.880985 19343 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:55573
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:54.915988 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36739
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:55.015759 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:55.015934 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:55.015986 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:55.019152 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:55.019212 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:55.019277 19543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:07:55.022922 19543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:55.023818 19543 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:55.024554 19543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:55.030550 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:55.030675 19543 server_base.cc:1047] running on GCE node
W20251025 20:07:55.030599 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:55.030529 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:55.031103 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:55.031603 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:55.032754 19543 hybrid_clock.cc:648] HybridClock initialized: now 1761422875032726 us; error 39 us; skew 500 ppm
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:55.035125 19543 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:07:55.036041 19543 webserver.cc:492] Webserver started at http://127.18.82.66:41041/ using document root <none> and password file <none>
I20251025 20:07:55.036521 19543 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:55.036571 19543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:55.036732 19543 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:55.038369 19543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "31102cfa3f2f4aa4989beb5dc4962f11"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "eede4ea42eb947f189092dd5c71c6a11"
server_key_iv: "379a46a5174f4fdd1536b7ab80d34c6a"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.038816 19543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "31102cfa3f2f4aa4989beb5dc4962f11"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "eede4ea42eb947f189092dd5c71c6a11"
server_key_iv: "379a46a5174f4fdd1536b7ab80d34c6a"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.041863 19543 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:07:55.043856 19558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.044714 19543 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:55.044813 19543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "31102cfa3f2f4aa4989beb5dc4962f11"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "eede4ea42eb947f189092dd5c71c6a11"
server_key_iv: "379a46a5174f4fdd1536b7ab80d34c6a"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.044889 19543 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:55.055464 19543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:55.057889 19543 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:55.058050 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:55.058559 19543 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:55.059315 19543 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:55.059361 19543 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.059422 19543 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:55.059451 19543 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.067972 19543 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:45497
I20251025 20:07:55.068006 19671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:45497 every 8 connection(s)
I20251025 20:07:55.068819 19543 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:07:55.071272 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19543
I20251025 20:07:55.071382 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:07:55.071646 18761 external_mini_cluster.cc:1468] Setting key c4f4648e04936ddba32307ffed36403b
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:07:55.081284 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.070436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39715 (local address 127.18.82.126:36739)
1025 20:07:55.070564 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.070568 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.071168 (+ 600us) server_negotiation.cc:408] Connection header received
1025 20:07:55.072224 (+ 1056us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.072228 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.072276 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.072351 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.074290 (+ 1939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.074734 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.075494 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.075687 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.078020 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.078037 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.078039 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.078066 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.079429 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.080026 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.080028 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.080029 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.080063 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.080311 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.080313 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.080314 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.080434 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.080549 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.081078 (+ 529us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.081168 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:07:55.082011 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.070654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36739 (local address 127.18.82.66:39715)
1025 20:07:55.071043 (+ 389us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.071081 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.072026 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.072498 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.072506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.072933 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:55.074142 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.074162 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.074830 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.074836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.075379 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.075388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.075554 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.076040 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:55.076061 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:55.077845 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:55.079581 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:55.079587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:55.079600 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:55.079934 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:55.080133 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:55.080138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:55.080142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:55.080242 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:55.080589 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:55.080597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:55.080875 (+ 278us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.081047 (+ 172us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.081237 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
I20251025 20:07:55.083125 19672 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36739
I20251025 20:07:55.083355 19672 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:55.083822 19672 heartbeater.cc:507] Master 127.18.82.126:36739 requested a full tablet report, sending...
I20251025 20:07:55.084857 19343 ts_manager.cc:194] Registered new tserver with Master: 31102cfa3f2f4aa4989beb5dc4962f11 (127.18.82.66:45497)
I20251025 20:07:55.085399 19343 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:39715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:55.115909 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36739
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:55.217850 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:55.218052 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:55.218133 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:55.221433 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:55.221490 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:55.221608 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:07:55.225294 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:55.226269 19679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:55.227118 19679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:55.233704 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:55.233709 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:55.233752 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:55.233848 19679 server_base.cc:1047] running on GCE node
I20251025 20:07:55.234206 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:55.234707 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:55.235847 19679 hybrid_clock.cc:648] HybridClock initialized: now 1761422875235828 us; error 39 us; skew 500 ppm
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:55.238351 19679 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:07:55.239284 19679 webserver.cc:492] Webserver started at http://127.18.82.67:33155/ using document root <none> and password file <none>
I20251025 20:07:55.239749 19679 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:55.239791 19679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:55.239936 19679 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:55.241494 19679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "128cce624916484d8f710b3af61f0d08"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "ec31ed5b5473914e67914970cbe95926"
server_key_iv: "87c3249d70da795776277c7950769bba"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.241966 19679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "128cce624916484d8f710b3af61f0d08"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "ec31ed5b5473914e67914970cbe95926"
server_key_iv: "87c3249d70da795776277c7950769bba"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.245136 19679 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251025 20:07:55.247232 19694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.248248 19679 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 20:07:55.248335 19679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "128cce624916484d8f710b3af61f0d08"
format_stamp: "Formatted at 2025-10-25 20:07:55 on dist-test-slave-d2pk"
server_key: "ec31ed5b5473914e67914970cbe95926"
server_key_iv: "87c3249d70da795776277c7950769bba"
server_key_version: "encryptionkey@0"
I20251025 20:07:55.248411 19679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:55.258838 19679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:55.261195 19679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:55.261327 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:55.261847 19679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:07:55.262599 19679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:55.262642 19679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.262676 19679 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:55.262696 19679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:55.271257 19679 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:36555
I20251025 20:07:55.271288 19807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:36555 every 8 connection(s)
I20251025 20:07:55.272133 19679 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:55.280959 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19679
I20251025 20:07:55.281068 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:07:55.281343 18761 external_mini_cluster.cc:1468] Setting key c61bc7717e59bb644dbb635ae1c3730c
I20251025 20:07:55.283766 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.273832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:59245 (local address 127.18.82.126:36739)
1025 20:07:55.273963 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.273966 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.274562 (+ 596us) server_negotiation.cc:408] Connection header received
1025 20:07:55.275366 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.275370 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.275431 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.275534 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.277167 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.277705 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.278371 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.278569 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.280538 (+ 1969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.280556 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.280559 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.280583 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.281990 (+ 1407us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.282460 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.282464 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.282465 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.282508 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.282761 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.282764 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.282766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.282916 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.283003 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.283488 (+ 485us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.283599 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 20:07:55.284276 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.274040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36739 (local address 127.18.82.67:59245)
1025 20:07:55.274413 (+ 373us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.274446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.275177 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.275695 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.275703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.276051 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:55.276967 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.276987 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.277826 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.277830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.278237 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.278268 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.278501 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.279072 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:55.279094 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:55.280402 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:55.282095 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:55.282101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:55.282111 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:55.282366 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:55.282584 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:55.282587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:55.282590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:55.282677 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:55.283014 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:55.283022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:55.283232 (+ 210us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.283440 (+ 208us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.283683 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
I20251025 20:07:55.285176 19808 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36739
I20251025 20:07:55.285370 19808 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:55.285830 19808 heartbeater.cc:507] Master 127.18.82.126:36739 requested a full tablet report, sending...
I20251025 20:07:55.286723 19343 ts_manager.cc:194] Registered new tserver with Master: 128cce624916484d8f710b3af61f0d08 (127.18.82.67:36555)
I20251025 20:07:55.287230 19343 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:59245
I20251025 20:07:55.294683 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:07:55.303408 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.297317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.18.82.126:36739)
1025 20:07:55.297458 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.297464 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.297616 (+ 152us) server_negotiation.cc:408] Connection header received
1025 20:07:55.297741 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.297745 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.297791 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.297875 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.298646 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.299097 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.299741 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.299944 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.300903 (+ 959us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.300920 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.300923 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.300946 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.302080 (+ 1134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.302430 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.302435 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.302437 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.302478 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.302680 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.302683 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.302685 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.302845 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.302963 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.303156 (+ 193us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.303261 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:07:55.307156 19343 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:07:55.309487 19343 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:07:55.321471 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:45497 (local address 127.0.0.1:53852)
1025 20:07:55.318405 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.318419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.318507 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.318941 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.318944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.318963 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:55.319207 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.319215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320165 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.320169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.321094 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.321101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.321180 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.321196 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.321240 (+ 44us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.321303 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":96,"threads_started":1}
I20251025 20:07:55.321755 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:37957 (local address 127.0.0.1:35488)
1025 20:07:55.318431 (+ 711us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.318445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.318524 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.319103 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.319106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.319130 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:55.319354 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.319360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320656 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.320659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.321372 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.321381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.321476 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.321492 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.321559 (+ 67us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.321610 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":99,"threads_started":1}
I20251025 20:07:55.322271 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.18.82.67:36555)
1025 20:07:55.317641 (+ 421us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.317648 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.318004 (+ 356us) server_negotiation.cc:408] Connection header received
1025 20:07:55.318098 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.318103 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.318294 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.318434 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:55.319051 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320145 (+ 1094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.321434 (+ 1289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.321963 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.322072 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.322120 (+ 48us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.322180 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":128,"threads_started":1}
I20251025 20:07:55.322274 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.18.82.66:45497)
1025 20:07:55.318200 (+ 409us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.318206 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.318603 (+ 397us) server_negotiation.cc:408] Connection header received
1025 20:07:55.318701 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.318705 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.318819 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.318937 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:55.319329 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320069 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.321339 (+ 1270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.321902 (+ 563us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.322009 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.322077 (+ 68us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.322155 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20251025 20:07:55.322751 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:36555 (local address 127.0.0.1:51552)
1025 20:07:55.317698 (+ 638us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.317737 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.317883 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.318604 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.318609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.318642 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:55.318923 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.318934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320243 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.320246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.320985 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.320995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.321090 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.321142 (+ 52us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.321224 (+ 82us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.321303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":82,"threads_started":1}
I20251025 20:07:55.322857 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.317902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.18.82.65:37957)
1025 20:07:55.318279 (+ 377us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.318285 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.318614 (+ 329us) server_negotiation.cc:408] Connection header received
1025 20:07:55.318701 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.318706 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.318869 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.318998 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:55.319713 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.320549 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.322003 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.322477 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.322572 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.322643 (+ 71us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.322725 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":114,"threads_started":1}
I20251025 20:07:55.324918 19606 tablet_service.cc:1505] Processing CreateTablet for tablet 31711b92909e4f8f866ff8fd3224744a (DEFAULT_TABLE table=test-table [id=30a880f1fc994296bfc4914352027638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:55.325038 19470 tablet_service.cc:1505] Processing CreateTablet for tablet 31711b92909e4f8f866ff8fd3224744a (DEFAULT_TABLE table=test-table [id=30a880f1fc994296bfc4914352027638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:55.325107 19742 tablet_service.cc:1505] Processing CreateTablet for tablet 31711b92909e4f8f866ff8fd3224744a (DEFAULT_TABLE table=test-table [id=30a880f1fc994296bfc4914352027638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:55.325882 19742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31711b92909e4f8f866ff8fd3224744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:55.325889 19470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31711b92909e4f8f866ff8fd3224744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:55.325889 19606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31711b92909e4f8f866ff8fd3224744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:55.331248 19825 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Bootstrap starting.
I20251025 20:07:55.332523 19827 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Bootstrap starting.
I20251025 20:07:55.333753 19826 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Bootstrap starting.
I20251025 20:07:55.333997 19825 tablet_bootstrap.cc:654] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:55.334532 19827 tablet_bootstrap.cc:654] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:55.334838 19825 log.cc:826] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:55.335295 19827 log.cc:826] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:55.335358 19826 tablet_bootstrap.cc:654] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:55.336054 19826 log.cc:826] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:55.336691 19825 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: No bootstrap required, opened a new log
I20251025 20:07:55.336906 19825 ts_tablet_manager.cc:1403] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251025 20:07:55.336920 19827 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: No bootstrap required, opened a new log
I20251025 20:07:55.337100 19827 ts_tablet_manager.cc:1403] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:07:55.337823 19826 tablet_bootstrap.cc:492] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: No bootstrap required, opened a new log
I20251025 20:07:55.338024 19826 ts_tablet_manager.cc:1403] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:07:55.339658 19825 raft_consensus.cc:359] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.339823 19825 raft_consensus.cc:385] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:55.339891 19825 raft_consensus.cc:740] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cce624916484d8f710b3af61f0d08, State: Initialized, Role: FOLLOWER
I20251025 20:07:55.340481 19825 consensus_queue.cc:260] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.340583 19827 raft_consensus.cc:359] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.340770 19827 raft_consensus.cc:385] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:55.340842 19827 raft_consensus.cc:740] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31102cfa3f2f4aa4989beb5dc4962f11, State: Initialized, Role: FOLLOWER
I20251025 20:07:55.340983 19826 raft_consensus.cc:359] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.341156 19826 raft_consensus.cc:385] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:55.341203 19826 raft_consensus.cc:740] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a917a5b676740a6a52300f1e9c678d9, State: Initialized, Role: FOLLOWER
I20251025 20:07:55.341239 19827 consensus_queue.cc:260] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.341310 19825 ts_tablet_manager.cc:1434] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:07:55.341678 19808 heartbeater.cc:499] Master 127.18.82.126:36739 was elected leader, sending a full tablet report...
I20251025 20:07:55.341954 19672 heartbeater.cc:499] Master 127.18.82.126:36739 was elected leader, sending a full tablet report...
I20251025 20:07:55.342038 19827 ts_tablet_manager.cc:1434] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 20:07:55.341583 19826 consensus_queue.cc:260] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.343384 19826 ts_tablet_manager.cc:1434] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:07:55.343591 19536 heartbeater.cc:499] Master 127.18.82.126:36739 was elected leader, sending a full tablet report...
W20251025 20:07:55.365167 19537 tablet.cc:2378] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:07:55.517735 19832 raft_consensus.cc:493] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:07:55.517915 19832 raft_consensus.cc:515] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.519064 19832 leader_election.cc:290] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128cce624916484d8f710b3af61f0d08 (127.18.82.67:36555), 4a917a5b676740a6a52300f1e9c678d9 (127.18.82.65:37957)
I20251025 20:07:55.522004 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.519296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:36555 (local address 127.18.82.66:60505)
1025 20:07:55.519424 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.519441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.519542 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.519901 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.519904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.519925 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:55.520168 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.520175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.521033 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.521037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.521716 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.521725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.521814 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.521832 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.521867 (+ 35us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.521910 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251025 20:07:55.522585 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.519372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:60505 (local address 127.18.82.67:36555)
1025 20:07:55.519511 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.519514 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.519531 (+ 17us) server_negotiation.cc:408] Connection header received
1025 20:07:55.519689 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.519692 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.519738 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.519826 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:55.520293 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.520922 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.521827 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.522281 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.522331 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.522409 (+ 78us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.522478 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 20:07:55.523010 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.519589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:37957 (local address 127.18.82.66:56329)
1025 20:07:55.519961 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:55.519975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:55.520051 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:55.520285 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:55.520288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:55.520306 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:55.520580 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.520586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.521311 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.521314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:55.521996 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:55.522006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.522739 (+ 733us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.522771 (+ 32us) client_negotiation.cc:769] Sending connection context
1025 20:07:55.522833 (+ 62us) client_negotiation.cc:240] Negotiation successful
1025 20:07:55.522892 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
W20251025 20:07:55.523016 19809 tablet.cc:2378] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:07:55.523113 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.519742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:56329 (local address 127.18.82.65:37957)
1025 20:07:55.519922 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.519926 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.520046 (+ 120us) server_negotiation.cc:408] Connection header received
1025 20:07:55.520137 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.520140 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.520188 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.520258 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:55.520689 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.521190 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.522143 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.522638 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.522677 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.522893 (+ 216us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.522979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251025 20:07:55.523227 19762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31711b92909e4f8f866ff8fd3224744a" candidate_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cce624916484d8f710b3af61f0d08" is_pre_election: true
I20251025 20:07:55.523512 19762 raft_consensus.cc:2468] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31102cfa3f2f4aa4989beb5dc4962f11 in term 0.
I20251025 20:07:55.523759 19490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31711b92909e4f8f866ff8fd3224744a" candidate_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a917a5b676740a6a52300f1e9c678d9" is_pre_election: true
I20251025 20:07:55.523990 19490 raft_consensus.cc:2468] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31102cfa3f2f4aa4989beb5dc4962f11 in term 0.
I20251025 20:07:55.523993 19560 leader_election.cc:304] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128cce624916484d8f710b3af61f0d08, 31102cfa3f2f4aa4989beb5dc4962f11; no voters:
I20251025 20:07:55.524266 19832 raft_consensus.cc:2804] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:07:55.524367 19832 raft_consensus.cc:493] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:07:55.524438 19832 raft_consensus.cc:3060] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:55.525316 19832 raft_consensus.cc:515] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.525684 19832 leader_election.cc:290] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [CANDIDATE]: Term 1 election: Requested vote from peers 128cce624916484d8f710b3af61f0d08 (127.18.82.67:36555), 4a917a5b676740a6a52300f1e9c678d9 (127.18.82.65:37957)
I20251025 20:07:55.525997 19762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31711b92909e4f8f866ff8fd3224744a" candidate_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cce624916484d8f710b3af61f0d08"
I20251025 20:07:55.526059 19490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31711b92909e4f8f866ff8fd3224744a" candidate_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a917a5b676740a6a52300f1e9c678d9"
I20251025 20:07:55.526111 19762 raft_consensus.cc:3060] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:55.526201 19490 raft_consensus.cc:3060] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:55.527376 19762 raft_consensus.cc:2468] T 31711b92909e4f8f866ff8fd3224744a P 128cce624916484d8f710b3af61f0d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31102cfa3f2f4aa4989beb5dc4962f11 in term 1.
I20251025 20:07:55.527491 19490 raft_consensus.cc:2468] T 31711b92909e4f8f866ff8fd3224744a P 4a917a5b676740a6a52300f1e9c678d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31102cfa3f2f4aa4989beb5dc4962f11 in term 1.
I20251025 20:07:55.527835 19560 leader_election.cc:304] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31102cfa3f2f4aa4989beb5dc4962f11, 4a917a5b676740a6a52300f1e9c678d9; no voters:
I20251025 20:07:55.528061 19832 raft_consensus.cc:2804] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:55.528308 19832 raft_consensus.cc:697] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [term 1 LEADER]: Becoming Leader. State: Replica: 31102cfa3f2f4aa4989beb5dc4962f11, State: Running, Role: LEADER
I20251025 20:07:55.528616 19832 consensus_queue.cc:237] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } }
I20251025 20:07:55.531598 19343 catalog_manager.cc:5649] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31102cfa3f2f4aa4989beb5dc4962f11 (127.18.82.66). New cstate: current_term: 1 leader_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cce624916484d8f710b3af61f0d08" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31102cfa3f2f4aa4989beb5dc4962f11" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 45497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a917a5b676740a6a52300f1e9c678d9" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37957 } health_report { overall_health: UNKNOWN } } }
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:55.556898 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.550143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.18.82.65:37957)
1025 20:07:55.550267 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.550272 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.550362 (+ 90us) server_negotiation.cc:408] Connection header received
1025 20:07:55.550479 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.550482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.550523 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.550583 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.551334 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.551766 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.552390 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.552598 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.554441 (+ 1843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.554463 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.554474 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.554497 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.555868 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.556176 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.556180 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.556181 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.556224 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.556350 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.556352 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.556353 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.556499 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.556633 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.556713 (+ 80us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.556796 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20251025 20:07:55.569715 19673 tablet.cc:2378] T 31711b92909e4f8f866ff8fd3224744a P 31102cfa3f2f4aa4989beb5dc4962f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:55.580516 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.573378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.18.82.65:37957)
1025 20:07:55.573557 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.573562 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.573618 (+ 56us) server_negotiation.cc:408] Connection header received
1025 20:07:55.573718 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.573721 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.573765 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.573832 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.574602 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.575010 (+ 408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.575611 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.575788 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.577750 (+ 1962us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.577766 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.577768 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.577793 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.579141 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.579537 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.579541 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.579542 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.579584 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.579805 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.579808 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.579810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.579929 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.580044 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.580255 (+ 211us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.580365 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:55.602501 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:55.595343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.18.82.65:37957)
1025 20:07:55.595485 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:55.595488 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:55.595532 (+ 44us) server_negotiation.cc:408] Connection header received
1025 20:07:55.595636 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:55.595639 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:55.595680 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:55.595742 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:55.596446 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.596899 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:55.597545 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:55.597766 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:55.599903 (+ 2137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:55.599920 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:55.599922 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:55.599945 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:55.601225 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.601673 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.601676 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.601677 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.601718 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:55.601926 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:55.601928 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:55.601930 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:55.602032 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:55.602118 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:55.602305 (+ 187us) server_negotiation.cc:300] Negotiation successful
1025 20:07:55.602382 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20251025 20:07:55.603694 19450 tablet_service.cc:3012] Rejecting scan request for tablet 31711b92909e4f8f866ff8fd3224744a: Uninitialized: clean time has not yet been initialized
I20251025 20:07:55.605289 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19407
I20251025 20:07:55.612732 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19543
I20251025 20:07:55.618481 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19679
I20251025 20:07:55.624255 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19311
2025-10-25T20:07:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3130 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19849](info): set up 2 sockets
Oct 25 20:07:55 dist-test-slave-d2pk krb5kdc[19849](info): commencing operation
krb5kdc: starting...
W20251025 20:07:57.458112 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:07:57 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:07:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:57.584271 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:42439
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:57.682559 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:57.682756 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:57.682808 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:57.685611 19865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:07:57.685657 19865 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 20:07:57.685673 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:57.685698 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:57.685731 19865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:07:57.685758 19865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:07:57.689769 19865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:42439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:42439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:57.690976 19865 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:57.691898 19865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:57.698329 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:57.698688 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:57.698329 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:57.698879 19865 server_base.cc:1047] running on GCE node
I20251025 20:07:57.699312 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:57.700080 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:57.701231 19865 hybrid_clock.cc:648] HybridClock initialized: now 1761422877701162 us; error 83 us; skew 500 ppm
Oct 25 20:07:57 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:57.703954 19865 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:07:57.705241 19865 webserver.cc:492] Webserver started at http://127.18.82.126:38795/ using document root <none> and password file <none>
I20251025 20:07:57.705960 19865 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:57.706013 19865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:57.706187 19865 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:57.708432 19865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "7eeb5075b91c412c87ee80bfcd4baca5"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "836b32593e4c30ae31d1510380686760"
server_key_iv: "070b92f335ec69fb4bd6e596ce7849fe"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.708950 19865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "7eeb5075b91c412c87ee80bfcd4baca5"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "836b32593e4c30ae31d1510380686760"
server_key_iv: "070b92f335ec69fb4bd6e596ce7849fe"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.712910 19865 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251025 20:07:57.715162 19880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:57.716034 19865 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:07:57.716135 19865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "7eeb5075b91c412c87ee80bfcd4baca5"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "836b32593e4c30ae31d1510380686760"
server_key_iv: "070b92f335ec69fb4bd6e596ce7849fe"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.716212 19865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:57.725375 19865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:57.728361 19865 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:57.728514 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:57.736016 19865 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:42439
I20251025 20:07:57.736207 19942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:42439 every 8 connection(s)
I20251025 20:07:57.736985 19865 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:07:57.739367 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19865
I20251025 20:07:57.739485 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:07:57.739719 18761 external_mini_cluster.cc:1468] Setting key a941187314661a841bfb7b29aa424d4a
I20251025 20:07:57.739873 19943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:57.745759 19943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Bootstrap starting.
Oct 25 20:07:57 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:57.748112 19943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:57.748932 19943 log.cc:826] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:57.750577 19943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: No bootstrap required, opened a new log
I20251025 20:07:57.751632 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:57.740858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.18.82.126:42439)
1025 20:07:57.741229 (+ 371us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:57.741243 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:57.741279 (+ 36us) server_negotiation.cc:408] Connection header received
1025 20:07:57.741961 (+ 682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:57.741985 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:57.742323 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:57.742652 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:57.743356 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:57.744439 (+ 1083us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:57.745005 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:57.745239 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:57.747490 (+ 2251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:57.747527 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:57.747540 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:57.747579 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:57.749499 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:57.749900 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:57.749905 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:57.749909 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:57.749973 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:57.750172 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:57.750175 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:57.750177 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:57.750469 (+ 292us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:57.750643 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:57.750897 (+ 254us) server_negotiation.cc:300] Negotiation successful
1025 20:07:57.751107 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20251025 20:07:57.753647 19943 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } }
I20251025 20:07:57.753810 19943 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:57.753865 19943 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeb5075b91c412c87ee80bfcd4baca5, State: Initialized, Role: FOLLOWER
I20251025 20:07:57.754228 19943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } }
I20251025 20:07:57.754341 19943 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:07:57.754405 19943 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:07:57.754489 19943 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:57.755332 19943 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } }
I20251025 20:07:57.755659 19943 leader_election.cc:304] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeb5075b91c412c87ee80bfcd4baca5; no voters:
I20251025 20:07:57.755915 19943 leader_election.cc:290] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:07:57.756000 19948 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:57.756212 19948 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [term 1 LEADER]: Becoming Leader. State: Replica: 7eeb5075b91c412c87ee80bfcd4baca5, State: Running, Role: LEADER
I20251025 20:07:57.756469 19948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } }
I20251025 20:07:57.756912 19943 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:07:57.757993 19949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } } }
I20251025 20:07:57.758111 19949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [sys.catalog]: This master's current role is: LEADER
I20251025 20:07:57.758476 19957 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:07:57.758445 19950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeb5075b91c412c87ee80bfcd4baca5. Latest consensus state: current_term: 1 leader_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb5075b91c412c87ee80bfcd4baca5" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42439 } } }
I20251025 20:07:57.758633 19950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5 [sys.catalog]: This master's current role is: LEADER
I20251025 20:07:57.760874 19957 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:07:57.765496 19957 catalog_manager.cc:1357] Generated new cluster ID: 73da3bff36044607a37740cdba6787a6
I20251025 20:07:57.765569 19957 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:07:57.773808 19957 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:07:57.774667 19957 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:07:57.782707 19957 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Generated new TSK 0
I20251025 20:07:57.783347 19957 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:57.833848 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42439
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:57.946611 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:57.946789 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:57.946846 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:57.949954 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:57.950004 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:57.950071 19971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:07:57.950142 19971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:07:57.953792 19971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:57.954713 19971 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:57.955492 19971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:57.961699 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:57.961721 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:57.961725 19977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:57.961896 19971 server_base.cc:1047] running on GCE node
I20251025 20:07:57.962216 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:57.962711 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:57.963871 19971 hybrid_clock.cc:648] HybridClock initialized: now 1761422877963854 us; error 26 us; skew 500 ppm
Oct 25 20:07:57 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:57.966192 19971 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:07:57.967283 19971 webserver.cc:492] Webserver started at http://127.18.82.65:41075/ using document root <none> and password file <none>
I20251025 20:07:57.967821 19971 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:57.967876 19971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:57.968053 19971 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:57.969673 19971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "e5f60abded3b4edda9c178d35c1b12dd"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "5a8be2413d4ba13bd67d8f74788306bd"
server_key_iv: "9afabfe8ad964ba2424247f84c48306d"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.970106 19971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "e5f60abded3b4edda9c178d35c1b12dd"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "5a8be2413d4ba13bd67d8f74788306bd"
server_key_iv: "9afabfe8ad964ba2424247f84c48306d"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.973131 19971 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 20:07:57.975128 19986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:57.976040 19971 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 20:07:57.976145 19971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "e5f60abded3b4edda9c178d35c1b12dd"
format_stamp: "Formatted at 2025-10-25 20:07:57 on dist-test-slave-d2pk"
server_key: "5a8be2413d4ba13bd67d8f74788306bd"
server_key_iv: "9afabfe8ad964ba2424247f84c48306d"
server_key_version: "encryptionkey@0"
I20251025 20:07:57.976226 19971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:57.994879 19971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:57.997606 19971 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:57.997782 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:57.999145 19971 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:57.999191 19971 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:57.999243 19971 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:57.999269 19971 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:58.010010 19971 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:37235
I20251025 20:07:58.010048 20101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:37235 every 8 connection(s)
I20251025 20:07:58.010710 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:57.999968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.65:40521)
1025 20:07:58.000369 (+ 401us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.000417 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.001114 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.001616 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.001624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.002070 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.002635 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.002648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.003631 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.003636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.004026 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.004033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.004245 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.004864 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.004895 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.006420 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.008412 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.008419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.008429 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.008658 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.008922 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.008925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.008927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.009003 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.009354 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.009375 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.009610 (+ 235us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.009794 (+ 184us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.010049 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20251025 20:07:58.010943 19971 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:07:58.011499 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:57.999714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:40521 (local address 127.18.82.126:42439)
1025 20:07:57.999892 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:57.999897 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.000515 (+ 618us) server_negotiation.cc:408] Connection header received
1025 20:07:58.001308 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.001312 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.001372 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.001466 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.002770 (+ 1304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.003507 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.004161 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.004336 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.006566 (+ 2230us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.006593 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.006597 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.006631 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.008289 (+ 1658us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.008752 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.008756 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.008758 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.008817 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.009096 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.009099 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.009101 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.009253 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.009342 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.011236 (+ 1894us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.011360 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251025 20:07:58.012493 20102 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42439
I20251025 20:07:58.012665 20102 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:58.013163 20102 heartbeater.cc:507] Master 127.18.82.126:42439 requested a full tablet report, sending...
I20251025 20:07:58.014600 19896 ts_manager.cc:194] Registered new tserver with Master: e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65:37235)
I20251025 20:07:58.016031 19896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:40521
I20251025 20:07:58.019403 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 19971
I20251025 20:07:58.019507 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:07:58.019796 18761 external_mini_cluster.cc:1468] Setting key 70a1c86b17618b11fc57a55e52a92c97
I20251025 20:07:58.020399 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.014309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.65:34223)
1025 20:07:58.014678 (+ 369us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.014693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.014812 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.015063 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.015065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.015200 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.015395 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.015400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.016264 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.016267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.016628 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.016635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.016716 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.017171 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.017185 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.017554 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.019116 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.019119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.019120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.019291 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.019575 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.019578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.019580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.019630 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.019979 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.019981 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.020050 (+ 69us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.020136 (+ 86us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.020245 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":79,"threads_started":1}
I20251025 20:07:58.020450 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.014376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:34223 (local address 127.18.82.126:42439)
1025 20:07:58.014513 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.014519 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.014755 (+ 236us) server_negotiation.cc:408] Connection header received
1025 20:07:58.014901 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.014905 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.014948 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.015040 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.015502 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.016168 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.016718 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.016887 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.017678 (+ 791us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.017693 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.017695 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.017715 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.019019 (+ 1304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.019389 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.019400 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.019402 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.019451 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.019732 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.019736 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.019737 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.019895 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.019988 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.020203 (+ 215us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.020297 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:58.060279 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42439
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:58.158746 20113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:58.158937 20113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:58.159003 20113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:58.161806 20113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:58.161850 20113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:58.161895 20113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:07:58.161932 20113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:07:58.165326 20113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:58.166355 20113 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:58.167178 20113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:58.172986 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:58.172992 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:58.172992 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:58.173243 20113 server_base.cc:1047] running on GCE node
I20251025 20:07:58.173581 20113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:58.174042 20113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:58.175194 20113 hybrid_clock.cc:648] HybridClock initialized: now 1761422878175172 us; error 34 us; skew 500 ppm
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:58.177495 20113 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:07:58.178498 20113 webserver.cc:492] Webserver started at http://127.18.82.66:45155/ using document root <none> and password file <none>
I20251025 20:07:58.178978 20113 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:58.179023 20113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:58.179195 20113 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:58.180773 20113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "04f5ed23454b42e88f8828edd920c14e"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "df90e9548b00f71819acd03ba2f49a88"
server_key_iv: "c89469433fd4556a9c494b47cc87964f"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.181195 20113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "04f5ed23454b42e88f8828edd920c14e"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "df90e9548b00f71819acd03ba2f49a88"
server_key_iv: "c89469433fd4556a9c494b47cc87964f"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.184429 20113 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20251025 20:07:58.186506 20128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:58.187474 20113 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:07:58.187572 20113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "04f5ed23454b42e88f8828edd920c14e"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "df90e9548b00f71819acd03ba2f49a88"
server_key_iv: "c89469433fd4556a9c494b47cc87964f"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.187672 20113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:58.223623 20113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:58.226835 20113 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:58.227008 20113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:58.228437 20113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:58.228487 20113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:58.228551 20113 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:58.228590 20113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:58.239667 20113 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:41191
I20251025 20:07:58.239707 20243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:41191 every 8 connection(s)
I20251025 20:07:58.240554 20113 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:07:58.240892 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.229088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:58015 (local address 127.18.82.126:42439)
1025 20:07:58.229222 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.229226 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.230040 (+ 814us) server_negotiation.cc:408] Connection header received
1025 20:07:58.231106 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.231110 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.231183 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.231256 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.232589 (+ 1333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.233226 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.234176 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.234384 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.237154 (+ 2770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.237173 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.237176 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.237201 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.239046 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.239523 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.239528 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.239531 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.239585 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.239919 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.239922 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.239923 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.240064 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.240155 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.240612 (+ 457us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.240733 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:07:58.241494 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.229417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.66:58015)
1025 20:07:58.229908 (+ 491us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.229941 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.230905 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.231444 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.231455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.231935 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.232448 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.232460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.233352 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.233359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.234049 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.234062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.234313 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.235004 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.235043 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.237000 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.239159 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.239164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.239172 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.239387 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.239696 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.239699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.239700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.239806 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.240168 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.240176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.240383 (+ 207us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.240561 (+ 178us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.240815 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20251025 20:07:58.242628 20244 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42439
I20251025 20:07:58.242775 20244 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:58.243248 20244 heartbeater.cc:507] Master 127.18.82.126:42439 requested a full tablet report, sending...
I20251025 20:07:58.244167 19896 ts_manager.cc:194] Registered new tserver with Master: 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191)
I20251025 20:07:58.244912 19896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:58015
I20251025 20:07:58.245409 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20113
I20251025 20:07:58.245553 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:07:58.245838 18761 external_mini_cluster.cc:1468] Setting key f5bac37ea12add323386fa1188deb0a2
I20251025 20:07:58.250839 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.244517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.66:40055)
1025 20:07:58.244862 (+ 345us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.244877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.244982 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.245344 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.245347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.245539 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.245781 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.245788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.246570 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.246577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.246985 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.246999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.247099 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.247678 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.247701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.247958 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.249593 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.249596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.249598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.249765 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.250025 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.250027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.250028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.250073 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.250451 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.250452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.250512 (+ 60us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.250584 (+ 72us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.250717 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20251025 20:07:58.250991 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.244596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:40055 (local address 127.18.82.126:42439)
1025 20:07:58.244718 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.244723 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.244948 (+ 225us) server_negotiation.cc:408] Connection header received
1025 20:07:58.245111 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.245114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.245167 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.245244 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.245889 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.246374 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.247085 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.247287 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.248068 (+ 781us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.248085 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.248088 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.248109 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.249483 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.249859 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.249865 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.249870 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.249920 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.250143 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.250148 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.250152 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.250360 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.250475 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.250658 (+ 183us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.250787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:07:58.289034 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42439
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:07:58.385617 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:07:58.385792 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:07:58.385874 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:07:58.388851 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:07:58.388900 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:07:58.388971 20255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:07:58.389031 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:07:58.392647 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:42439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:07:58.393582 20255 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:07:58.394312 20255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:07:58.399928 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:07:58.399906 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:58.400089 20255 server_base.cc:1047] running on GCE node
W20251025 20:07:58.399914 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:07:58.400519 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:07:58.400943 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:07:58.402078 20255 hybrid_clock.cc:648] HybridClock initialized: now 1761422878402065 us; error 27 us; skew 500 ppm
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:07:58.404471 20255 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:07:58.405398 20255 webserver.cc:492] Webserver started at http://127.18.82.67:44277/ using document root <none> and password file <none>
I20251025 20:07:58.405891 20255 fs_manager.cc:362] Metadata directory not provided
I20251025 20:07:58.405937 20255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:07:58.406083 20255 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:07:58.407686 20255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "116ab0ab9b04415fba8a95256498bc33"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "6cae8435bfe364d7651267ae785167ec"
server_key_iv: "2c9921b2dd6bb8eb6b9aec3106da903a"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.408105 20255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "116ab0ab9b04415fba8a95256498bc33"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "6cae8435bfe364d7651267ae785167ec"
server_key_iv: "2c9921b2dd6bb8eb6b9aec3106da903a"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.411075 20255 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:07:58.412997 20270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:58.413904 20255 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:58.413988 20255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "116ab0ab9b04415fba8a95256498bc33"
format_stamp: "Formatted at 2025-10-25 20:07:58 on dist-test-slave-d2pk"
server_key: "6cae8435bfe364d7651267ae785167ec"
server_key_iv: "2c9921b2dd6bb8eb6b9aec3106da903a"
server_key_version: "encryptionkey@0"
I20251025 20:07:58.414053 20255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:07:58.431879 20255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:07:58.434561 20255 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:07:58.434696 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:07:58.435930 20255 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:07:58.435976 20255 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:07:58.436033 20255 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:07:58.436072 20255 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:58.445909 20255 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34563
I20251025 20:07:58.445974 20385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34563 every 8 connection(s)
I20251025 20:07:58.446975 20255 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:07:58.447317 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.436519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:42413 (local address 127.18.82.126:42439)
1025 20:07:58.436649 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.436653 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.437365 (+ 712us) server_negotiation.cc:408] Connection header received
1025 20:07:58.438253 (+ 888us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.438257 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.438324 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.438400 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.439891 (+ 1491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.440637 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.441470 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.441725 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.443599 (+ 1874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.443620 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.443625 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.443693 (+ 68us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.445305 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.445854 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.445857 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.445858 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.445913 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.446217 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.446221 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.446222 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.446443 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.446553 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.447022 (+ 469us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.447140 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:07:58.447845 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.436804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.67:42413)
1025 20:07:58.437206 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.437237 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.438057 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.438582 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.438589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.439051 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.439736 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.439749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.440752 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.440756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.441340 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.441351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.441595 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.442142 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.442167 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.443461 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.445407 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.445412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.445421 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.445743 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.446012 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.446015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.446017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.446111 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.446538 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.446543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.446761 (+ 218us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.446908 (+ 147us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.447126 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20251025 20:07:58.448992 20386 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42439
I20251025 20:07:58.449131 20386 heartbeater.cc:461] Registering TS with master...
I20251025 20:07:58.449676 20386 heartbeater.cc:507] Master 127.18.82.126:42439 requested a full tablet report, sending...
I20251025 20:07:58.450580 19897 ts_manager.cc:194] Registered new tserver with Master: 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.451161 19897 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:42413
I20251025 20:07:58.454066 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20255
I20251025 20:07:58.454182 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:07:58.454403 18761 external_mini_cluster.cc:1468] Setting key 4684ae1f95c94efd4f384d84527b4dc6
I20251025 20:07:58.456202 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:07:58.456414 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.450927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:42935 (local address 127.18.82.126:42439)
1025 20:07:58.451024 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.451028 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.451119 (+ 91us) server_negotiation.cc:408] Connection header received
1025 20:07:58.451199 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.451202 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.451244 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.451317 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.451807 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.452417 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.452953 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.453086 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.453755 (+ 669us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.453769 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.453771 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.453792 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.455177 (+ 1385us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.455535 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.455538 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.455539 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.455582 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.455772 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.455775 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.455777 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.455917 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.456005 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.456221 (+ 216us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.456317 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:07:58.456411 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.450857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.18.82.67:42935)
1025 20:07:58.451057 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.451067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.451135 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.451335 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.451339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.451508 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.451704 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.451709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.452503 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.452505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.452838 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.452844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.452931 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.453353 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.453364 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.453653 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.455267 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.455271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.455272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.455450 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.455663 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.455665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.455667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.455707 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.455998 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.456003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.456083 (+ 80us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.456169 (+ 86us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.456277 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:58.477421 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.470970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.18.82.126:42439)
1025 20:07:58.471073 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.471076 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.471086 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:07:58.471116 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.471119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.471151 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.471212 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.471866 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.472298 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.472837 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.472946 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.475268 (+ 2322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.475284 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.475286 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.475321 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.476402 (+ 1081us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.476705 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.476707 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.476708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.476737 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.476869 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.476871 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.476872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.476978 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.477064 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.477173 (+ 109us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.477247 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:07:58.479782 19897 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:07:58.482080 19897 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:07:58.493211 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.489577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:41191 (local address 127.0.0.1:53904)
1025 20:07:58.489892 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.489931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.490055 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.490707 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.490712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.490733 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.490978 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.490988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.492217 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.492221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.492750 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.492757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.492856 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.492910 (+ 54us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.493006 (+ 96us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.493087 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20251025 20:07:58.493927 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.489690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.18.82.66:41191)
1025 20:07:58.490047 (+ 357us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.490053 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.490075 (+ 22us) server_negotiation.cc:408] Connection header received
1025 20:07:58.490179 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.490183 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.490363 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.490522 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.491114 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.492111 (+ 997us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.492887 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.493547 (+ 660us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.493666 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.493727 (+ 61us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.493805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":93,"threads_started":1}
I20251025 20:07:58.494271 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.490484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:34563 (local address 127.0.0.1:46482)
1025 20:07:58.490964 (+ 480us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.490976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.491050 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.491660 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.491662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.491688 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.491871 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.491877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.493217 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.493221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.493969 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.493980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.494067 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.494082 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.494114 (+ 32us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.494183 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":67,"threads_started":1}
I20251025 20:07:58.494801 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.490182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:37235 (local address 127.0.0.1:56538)
1025 20:07:58.490418 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.490439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.490525 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.491613 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.491616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.491635 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.491866 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.491872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.493703 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.493706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.494508 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.494519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.494604 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.494619 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.494678 (+ 59us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.494722 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20251025 20:07:58.495378 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.490939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.18.82.67:34563)
1025 20:07:58.491264 (+ 325us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.491269 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.491284 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:07:58.491357 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.491361 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.491468 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.491990 (+ 522us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.492130 (+ 140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.493092 (+ 962us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.494564 (+ 1472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.495028 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.495129 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.495187 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.495257 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20251025 20:07:58.495797 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.490805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.18.82.65:37235)
1025 20:07:58.491164 (+ 359us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.491170 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.491190 (+ 20us) server_negotiation.cc:408] Connection header received
1025 20:07:58.491266 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.491271 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.491469 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.491602 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.491979 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.493010 (+ 1031us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.495116 (+ 2106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.495503 (+ 387us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.495590 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.495648 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.495711 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":66,"threads_started":1}
I20251025 20:07:58.496786 20178 tablet_service.cc:1505] Processing CreateTablet for tablet c28c5e55b13c4174b433c3e84fee4011 (DEFAULT_TABLE table=test-table [id=16460cae93754b12a3408942aea483be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.497786 20178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28c5e55b13c4174b433c3e84fee4011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.497700 20320 tablet_service.cc:1505] Processing CreateTablet for tablet c28c5e55b13c4174b433c3e84fee4011 (DEFAULT_TABLE table=test-table [id=16460cae93754b12a3408942aea483be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.497761 20036 tablet_service.cc:1505] Processing CreateTablet for tablet c28c5e55b13c4174b433c3e84fee4011 (DEFAULT_TABLE table=test-table [id=16460cae93754b12a3408942aea483be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.498528 20320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28c5e55b13c4174b433c3e84fee4011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.498634 20036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28c5e55b13c4174b433c3e84fee4011. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.503077 20409 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Bootstrap starting.
I20251025 20:07:58.504384 20411 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Bootstrap starting.
I20251025 20:07:58.505066 20409 tablet_bootstrap.cc:654] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.505821 20410 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Bootstrap starting.
I20251025 20:07:58.506009 20411 tablet_bootstrap.cc:654] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.506271 20409 log.cc:826] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:58.507121 20411 log.cc:826] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:58.507694 20410 tablet_bootstrap.cc:654] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.508308 20409 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: No bootstrap required, opened a new log
I20251025 20:07:58.508513 20409 ts_tablet_manager.cc:1403] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20251025 20:07:58.508836 20410 log.cc:826] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Log is configured to *not* fsync() on all Append() calls
I20251025 20:07:58.508862 20411 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: No bootstrap required, opened a new log
I20251025 20:07:58.509013 20411 ts_tablet_manager.cc:1403] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:07:58.510471 20410 tablet_bootstrap.cc:492] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: No bootstrap required, opened a new log
I20251025 20:07:58.510643 20410 ts_tablet_manager.cc:1403] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:07:58.511396 20411 raft_consensus.cc:359] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.511570 20411 raft_consensus.cc:385] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.511457 20409 raft_consensus.cc:359] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.511620 20411 raft_consensus.cc:740] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116ab0ab9b04415fba8a95256498bc33, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.511632 20409 raft_consensus.cc:385] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.511698 20409 raft_consensus.cc:740] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f60abded3b4edda9c178d35c1b12dd, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.512030 20411 consensus_queue.cc:260] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.512115 20409 consensus_queue.cc:260] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.512811 20411 ts_tablet_manager.cc:1434] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:07:58.512845 20409 ts_tablet_manager.cc:1434] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20251025 20:07:58.513139 20386 heartbeater.cc:499] Master 127.18.82.126:42439 was elected leader, sending a full tablet report...
I20251025 20:07:58.513172 20102 heartbeater.cc:499] Master 127.18.82.126:42439 was elected leader, sending a full tablet report...
I20251025 20:07:58.513444 20410 raft_consensus.cc:359] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.513661 20410 raft_consensus.cc:385] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.513717 20410 raft_consensus.cc:740] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5ed23454b42e88f8828edd920c14e, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.514086 20410 consensus_queue.cc:260] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.514846 20410 ts_tablet_manager.cc:1434] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 20:07:58.515166 20244 heartbeater.cc:499] Master 127.18.82.126:42439 was elected leader, sending a full tablet report...
I20251025 20:07:58.548187 20416 raft_consensus.cc:493] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:07:58.548338 20416 raft_consensus.cc:515] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.549322 20416 leader_election.cc:290] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.552901 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.549535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:41191 (local address 127.18.82.65:43413)
1025 20:07:58.549850 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.549862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.549967 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.550230 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.550233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.550248 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.550500 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.550507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.551517 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.551520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.552496 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.552502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.552605 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.552626 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.552677 (+ 51us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.552742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":81,"threads_started":1}
I20251025 20:07:58.552906 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.549573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:34563 (local address 127.18.82.65:52085)
1025 20:07:58.549870 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.549886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.550079 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.550331 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.550334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.550348 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.550569 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.550577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.551517 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.551520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.552481 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.552493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.552605 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.552626 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.552683 (+ 57us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.552742 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20251025 20:07:58.553263 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.549621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:43413 (local address 127.18.82.66:41191)
1025 20:07:58.549746 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.549751 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.549943 (+ 192us) server_negotiation.cc:408] Connection header received
1025 20:07:58.550090 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.550092 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.550140 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.550212 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.550638 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.551402 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.552597 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.553038 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.553072 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.553115 (+ 43us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.553163 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:07:58.553263 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.549670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:52085 (local address 127.18.82.67:34563)
1025 20:07:58.549793 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.549798 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.549975 (+ 177us) server_negotiation.cc:408] Connection header received
1025 20:07:58.550188 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.550193 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.550241 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.550309 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.550689 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.551402 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.552597 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.553038 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.553072 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.553115 (+ 43us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.553163 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 20:07:58.553812 20198 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c28c5e55b13c4174b433c3e84fee4011" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5ed23454b42e88f8828edd920c14e" is_pre_election: true
I20251025 20:07:58.553822 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c28c5e55b13c4174b433c3e84fee4011" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33" is_pre_election: true
I20251025 20:07:58.554029 20340 raft_consensus.cc:2468] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 0.
I20251025 20:07:58.554034 20198 raft_consensus.cc:2468] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 0.
I20251025 20:07:58.554437 19988 leader_election.cc:304] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5ed23454b42e88f8828edd920c14e, e5f60abded3b4edda9c178d35c1b12dd; no voters:
I20251025 20:07:58.554649 20416 raft_consensus.cc:2804] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:07:58.554713 20416 raft_consensus.cc:493] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:07:58.554749 20416 raft_consensus.cc:3060] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.555527 20416 raft_consensus.cc:515] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.555817 20416 leader_election.cc:290] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 election: Requested vote from peers 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.556118 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c28c5e55b13c4174b433c3e84fee4011" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33"
I20251025 20:07:58.556226 20340 raft_consensus.cc:3060] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.556259 20198 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c28c5e55b13c4174b433c3e84fee4011" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5ed23454b42e88f8828edd920c14e"
I20251025 20:07:58.556371 20198 raft_consensus.cc:3060] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.557123 20340 raft_consensus.cc:2468] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 1.
I20251025 20:07:58.557328 20198 raft_consensus.cc:2468] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 1.
I20251025 20:07:58.557472 19989 leader_election.cc:304] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116ab0ab9b04415fba8a95256498bc33, e5f60abded3b4edda9c178d35c1b12dd; no voters:
I20251025 20:07:58.557672 20416 raft_consensus.cc:2804] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:58.557866 20416 raft_consensus.cc:697] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 1 LEADER]: Becoming Leader. State: Replica: e5f60abded3b4edda9c178d35c1b12dd, State: Running, Role: LEADER
I20251025 20:07:58.558164 20416 consensus_queue.cc:237] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.560997 19896 catalog_manager.cc:5649] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd reported cstate change: term changed from 0 to 1, leader changed from <none> to e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65). New cstate: current_term: 1 leader_uuid: "e5f60abded3b4edda9c178d35c1b12dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20251025 20:07:58.582616 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.579733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.18.82.65:37235)
1025 20:07:58.579859 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.579862 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.579876 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:07:58.580043 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.580045 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.580095 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.580168 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:07:58.580492 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.580990 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.581699 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.581844 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.581894 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.582311 (+ 417us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.582399 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.582477 (+ 78us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.582521 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:07:58.587852 20340 raft_consensus.cc:1275] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Refusing update from remote peer e5f60abded3b4edda9c178d35c1b12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.587852 20198 raft_consensus.cc:1275] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Refusing update from remote peer e5f60abded3b4edda9c178d35c1b12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.588485 20420 consensus_queue.cc:1048] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.588650 20416 consensus_queue.cc:1048] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.594753 20423 mvcc.cc:204] Tried to move back new op lower bound from 7214788110689153024 to 7214788110576877568. Current Snapshot: MvccSnapshot[applied={T|T < 7214788110689153024}]
I20251025 20:07:58.595378 20425 mvcc.cc:204] Tried to move back new op lower bound from 7214788110689153024 to 7214788110576877568. Current Snapshot: MvccSnapshot[applied={T|T < 7214788110689153024}]
I20251025 20:07:58.596777 20424 mvcc.cc:204] Tried to move back new op lower bound from 7214788110689153024 to 7214788110576877568. Current Snapshot: MvccSnapshot[applied={T|T < 7214788110689153024}]
I20251025 20:07:58.605307 19897 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:07:58.605468 19897 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:07:58.607861 19897 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Sending DeleteTablet for 3 replicas of tablet c28c5e55b13c4174b433c3e84fee4011
I20251025 20:07:58.608652 20036 tablet_service.cc:1552] Processing DeleteTablet for tablet c28c5e55b13c4174b433c3e84fee4011 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:56538
I20251025 20:07:58.608700 20178 tablet_service.cc:1552] Processing DeleteTablet for tablet c28c5e55b13c4174b433c3e84fee4011 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:53904
I20251025 20:07:58.608700 20320 tablet_service.cc:1552] Processing DeleteTablet for tablet c28c5e55b13c4174b433c3e84fee4011 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:46482
I20251025 20:07:58.609119 20435 tablet_replica.cc:333] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: stopping tablet replica
I20251025 20:07:58.609380 20435 raft_consensus.cc:2243] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:07:58.609537 20433 tablet_replica.cc:333] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: stopping tablet replica
I20251025 20:07:58.609730 20435 raft_consensus.cc:2272] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:07:58.609776 20433 raft_consensus.cc:2243] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:07:58.609827 19897 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 20:07:58.610042 20434 tablet_replica.cc:333] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: stopping tablet replica
I20251025 20:07:58.610111 20433 raft_consensus.cc:2272] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 20:07:58.610184 19897 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:07:58.610239 20434 raft_consensus.cc:2243] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:07:58.610469 20434 raft_consensus.cc:2272] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:07:58.612226 20435 ts_tablet_manager.cc:1916] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.612226 20433 ts_tablet_manager.cc:1916] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.614459 20434 ts_tablet_manager.cc:1916] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.615891 20178 tablet_service.cc:1505] Processing CreateTablet for tablet f5da69042d85418b919aba70c90b618e (DEFAULT_TABLE table=test-table [id=b6705afcc9af4406b7a89ed103e34043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.616146 20178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da69042d85418b919aba70c90b618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.616529 20435 ts_tablet_manager.cc:1929] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:07:58.616685 20435 log.cc:1199] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/c28c5e55b13c4174b433c3e84fee4011
I20251025 20:07:58.616935 20434 ts_tablet_manager.cc:1929] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:07:58.617024 20434 log.cc:1199] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/c28c5e55b13c4174b433c3e84fee4011
I20251025 20:07:58.617364 20434 ts_tablet_manager.cc:1950] T c28c5e55b13c4174b433c3e84fee4011 P 116ab0ab9b04415fba8a95256498bc33: Deleting consensus metadata
I20251025 20:07:58.618072 20435 ts_tablet_manager.cc:1950] T c28c5e55b13c4174b433c3e84fee4011 P 04f5ed23454b42e88f8828edd920c14e: Deleting consensus metadata
I20251025 20:07:58.618424 20410 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Bootstrap starting.
I20251025 20:07:58.618623 20036 tablet_service.cc:1505] Processing CreateTablet for tablet f5da69042d85418b919aba70c90b618e (DEFAULT_TABLE table=test-table [id=b6705afcc9af4406b7a89ed103e34043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.618854 20036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da69042d85418b919aba70c90b618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.619287 20410 tablet_bootstrap.cc:654] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.620637 20410 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: No bootstrap required, opened a new log
I20251025 20:07:58.620725 20410 ts_tablet_manager.cc:1403] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:58.620880 20409 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Bootstrap starting.
I20251025 20:07:58.621168 20410 raft_consensus.cc:359] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.621284 20410 raft_consensus.cc:385] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.621323 20410 raft_consensus.cc:740] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5ed23454b42e88f8828edd920c14e, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.621475 20410 consensus_queue.cc:260] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.621747 20410 ts_tablet_manager.cc:1434] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:07:58.621869 20409 tablet_bootstrap.cc:654] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.622684 19882 catalog_manager.cc:4985] TS 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191): tablet c28c5e55b13c4174b433c3e84fee4011 (table test-table [id=16460cae93754b12a3408942aea483be]) successfully deleted
I20251025 20:07:58.622730 19883 catalog_manager.cc:4985] TS 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563): tablet c28c5e55b13c4174b433c3e84fee4011 (table test-table [id=16460cae93754b12a3408942aea483be]) successfully deleted
I20251025 20:07:58.623106 20320 tablet_service.cc:1505] Processing CreateTablet for tablet f5da69042d85418b919aba70c90b618e (DEFAULT_TABLE table=test-table [id=b6705afcc9af4406b7a89ed103e34043]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:07:58.623342 20320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da69042d85418b919aba70c90b618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.625216 20411 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Bootstrap starting.
I20251025 20:07:58.625852 20409 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: No bootstrap required, opened a new log
I20251025 20:07:58.625937 20409 ts_tablet_manager.cc:1403] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251025 20:07:58.626250 20411 tablet_bootstrap.cc:654] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.626482 20409 raft_consensus.cc:359] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.626617 20409 raft_consensus.cc:385] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.626659 20409 raft_consensus.cc:740] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f60abded3b4edda9c178d35c1b12dd, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.626791 20409 consensus_queue.cc:260] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.627030 20409 ts_tablet_manager.cc:1434] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:07:58.628055 20433 ts_tablet_manager.cc:1929] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:07:58.628145 20433 log.cc:1199] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/c28c5e55b13c4174b433c3e84fee4011
I20251025 20:07:58.628446 20433 ts_tablet_manager.cc:1950] T c28c5e55b13c4174b433c3e84fee4011 P e5f60abded3b4edda9c178d35c1b12dd: Deleting consensus metadata
I20251025 20:07:58.629413 19883 catalog_manager.cc:4985] TS e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65:37235): tablet c28c5e55b13c4174b433c3e84fee4011 (table test-table [id=16460cae93754b12a3408942aea483be]) successfully deleted
I20251025 20:07:58.629901 20411 tablet_bootstrap.cc:492] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: No bootstrap required, opened a new log
I20251025 20:07:58.629988 20411 ts_tablet_manager.cc:1403] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251025 20:07:58.630340 20411 raft_consensus.cc:359] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.630429 20411 raft_consensus.cc:385] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.630462 20411 raft_consensus.cc:740] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116ab0ab9b04415fba8a95256498bc33, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.630590 20411 consensus_queue.cc:260] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.630841 20411 ts_tablet_manager.cc:1434] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251025 20:07:58.698100 20387 tablet.cc:2378] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:07:58.704241 20428 raft_consensus.cc:493] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:07:58.704352 20428 raft_consensus.cc:515] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.704797 20428 leader_election.cc:290] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.705183 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5da69042d85418b919aba70c90b618e" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33" is_pre_election: true
I20251025 20:07:58.705199 20198 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5da69042d85418b919aba70c90b618e" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5ed23454b42e88f8828edd920c14e" is_pre_election: true
I20251025 20:07:58.705314 20340 raft_consensus.cc:2468] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 0.
I20251025 20:07:58.705343 20198 raft_consensus.cc:2468] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 0.
I20251025 20:07:58.705602 19989 leader_election.cc:304] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116ab0ab9b04415fba8a95256498bc33, e5f60abded3b4edda9c178d35c1b12dd; no voters:
I20251025 20:07:58.705864 20428 raft_consensus.cc:2804] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:07:58.705916 20428 raft_consensus.cc:493] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:07:58.705957 20428 raft_consensus.cc:3060] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.706940 20428 raft_consensus.cc:515] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.707253 20428 leader_election.cc:290] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 election: Requested vote from peers 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.707595 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5da69042d85418b919aba70c90b618e" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33"
I20251025 20:07:58.707720 20340 raft_consensus.cc:3060] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.707741 20198 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5da69042d85418b919aba70c90b618e" candidate_uuid: "e5f60abded3b4edda9c178d35c1b12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f5ed23454b42e88f8828edd920c14e"
I20251025 20:07:58.707860 20198 raft_consensus.cc:3060] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.708557 20340 raft_consensus.cc:2468] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 1.
I20251025 20:07:58.708842 19989 leader_election.cc:304] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116ab0ab9b04415fba8a95256498bc33, e5f60abded3b4edda9c178d35c1b12dd; no voters:
I20251025 20:07:58.708894 20198 raft_consensus.cc:2468] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f60abded3b4edda9c178d35c1b12dd in term 1.
I20251025 20:07:58.708998 20428 raft_consensus.cc:2804] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:58.709082 20428 raft_consensus.cc:697] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 1 LEADER]: Becoming Leader. State: Replica: e5f60abded3b4edda9c178d35c1b12dd, State: Running, Role: LEADER
I20251025 20:07:58.709218 20428 consensus_queue.cc:237] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.710827 19897 catalog_manager.cc:5649] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd reported cstate change: term changed from 0 to 1, leader changed from <none> to e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65). New cstate: current_term: 1 leader_uuid: "e5f60abded3b4edda9c178d35c1b12dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } health_report { overall_health: UNKNOWN } } }
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:07:58.739252 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.731433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.18.82.126:42439)
1025 20:07:58.731567 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.731571 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.731752 (+ 181us) server_negotiation.cc:408] Connection header received
1025 20:07:58.731887 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.731890 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.731931 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.731997 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.732994 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.733449 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.734254 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.734372 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.736447 (+ 2075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.736461 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.736463 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.736483 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.737843 (+ 1360us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.738250 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.738256 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.738257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.738295 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.738520 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.738523 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.738525 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.738664 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.738745 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.738990 (+ 245us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.739145 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":52}
I20251025 20:07:58.739253 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.731350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42439 (local address 127.0.0.1:57576)
1025 20:07:58.731657 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.731674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.731797 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.732073 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.732077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.732622 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:07:58.732886 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.732897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.733575 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.733579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.734140 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.734150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.734269 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.734805 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:07:58.734821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:07:58.736323 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:07:58.737948 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.737956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.737961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.738157 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.738389 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:07:58.738391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:07:58.738393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:07:58.738434 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:07:58.738763 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:07:58.738770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:07:58.738851 (+ 81us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.738943 (+ 92us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.739082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
W20251025 20:07:58.741681 20245 tablet.cc:2378] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:07:58.743492 19896 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:57576:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 20:07:58.744130 19896 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:07:58.749053 20036 tablet_service.cc:1505] Processing CreateTablet for tablet a8db67fc1c1745caa4ad40cefd3009f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cba2ffa51c04f53ba796956a2938838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:07:58.749420 20036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8db67fc1c1745caa4ad40cefd3009f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.751609 20409 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: Bootstrap starting.
I20251025 20:07:58.751894 20178 tablet_service.cc:1505] Processing CreateTablet for tablet a8db67fc1c1745caa4ad40cefd3009f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cba2ffa51c04f53ba796956a2938838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:07:58.752245 20178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8db67fc1c1745caa4ad40cefd3009f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.752687 20409 tablet_bootstrap.cc:654] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.753165 20320 tablet_service.cc:1505] Processing CreateTablet for tablet a8db67fc1c1745caa4ad40cefd3009f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cba2ffa51c04f53ba796956a2938838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:07:58.753501 20320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8db67fc1c1745caa4ad40cefd3009f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:07:58.754832 20410 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: Bootstrap starting.
I20251025 20:07:58.755447 20409 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: No bootstrap required, opened a new log
I20251025 20:07:58.755519 20409 ts_tablet_manager.cc:1403] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251025 20:07:58.755775 20410 tablet_bootstrap.cc:654] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.755999 20409 raft_consensus.cc:359] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.756100 20409 raft_consensus.cc:385] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.756130 20409 raft_consensus.cc:740] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f60abded3b4edda9c178d35c1b12dd, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.756246 20409 consensus_queue.cc:260] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.756522 20409 ts_tablet_manager.cc:1434] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:07:58.756726 20411 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: Bootstrap starting.
I20251025 20:07:58.756532 20428 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.757242 20428 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.757756 20411 tablet_bootstrap.cc:654] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: Neither blocks nor log segments found. Creating new log.
I20251025 20:07:58.758332 20410 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: No bootstrap required, opened a new log
I20251025 20:07:58.758416 20410 ts_tablet_manager.cc:1403] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251025 20:07:58.758941 20410 raft_consensus.cc:359] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.759039 20410 raft_consensus.cc:385] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.759052 20411 tablet_bootstrap.cc:492] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: No bootstrap required, opened a new log
I20251025 20:07:58.759073 20410 raft_consensus.cc:740] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f5ed23454b42e88f8828edd920c14e, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.759120 20411 ts_tablet_manager.cc:1403] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:07:58.759184 20410 consensus_queue.cc:260] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.759433 20410 ts_tablet_manager.cc:1434] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:07:58.759517 20417 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.759594 20411 raft_consensus.cc:359] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.759627 20417 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.759667 20411 raft_consensus.cc:385] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:07:58.759687 20411 raft_consensus.cc:740] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116ab0ab9b04415fba8a95256498bc33, State: Initialized, Role: FOLLOWER
I20251025 20:07:58.759783 20411 consensus_queue.cc:260] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.760006 20411 ts_tablet_manager.cc:1434] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:07:58.759982 20415 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.760082 20415 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: This TxnStatusTablet replica's current role is: FOLLOWER
W20251025 20:07:58.762480 20103 tablet.cc:2378] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:07:58.874128 20417 raft_consensus.cc:493] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:07:58.874272 20417 raft_consensus.cc:515] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.875370 20417 leader_election.cc:290] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65:37235), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.878832 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.875903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:34563 (local address 127.18.82.66:34905)
1025 20:07:58.876149 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.876160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.876269 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.876489 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.876492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.876507 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.876721 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.876727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.877547 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.877550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.878374 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.878384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.878504 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.878529 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.878603 (+ 74us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.878663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":7680,"thread_start_us":66,"threads_started":1}
I20251025 20:07:58.879204 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.876006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:34905 (local address 127.18.82.67:34563)
1025 20:07:58.876148 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.876154 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.876218 (+ 64us) server_negotiation.cc:408] Connection header received
1025 20:07:58.876364 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.876366 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.876409 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.876465 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.876811 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.877439 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.878494 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.878970 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.879004 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.879053 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.879099 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:07:58.879204 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.875703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:41319 (local address 127.18.82.65:37235)
1025 20:07:58.875837 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.875843 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.875916 (+ 73us) server_negotiation.cc:408] Connection header received
1025 20:07:58.876062 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.876067 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.876118 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.876191 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:07:58.876663 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.877301 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.878502 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.878969 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.879004 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.879052 (+ 48us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.879106 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:07:58.878830 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.875566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:37235 (local address 127.18.82.66:41319)
1025 20:07:58.875830 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.875845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.875960 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.876222 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.876226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.876262 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:07:58.876540 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.876549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.877428 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.877431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.878374 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.878384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.878515 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.878529 (+ 14us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.878603 (+ 74us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.878663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"spinlock_wait_cycles":14848,"thread_start_us":90,"threads_started":1}
I20251025 20:07:58.879601 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8db67fc1c1745caa4ad40cefd3009f7" candidate_uuid: "04f5ed23454b42e88f8828edd920c14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33" is_pre_election: true
I20251025 20:07:58.879752 20340 raft_consensus.cc:2468] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f5ed23454b42e88f8828edd920c14e in term 0.
I20251025 20:07:58.879952 20056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8db67fc1c1745caa4ad40cefd3009f7" candidate_uuid: "04f5ed23454b42e88f8828edd920c14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f60abded3b4edda9c178d35c1b12dd" is_pre_election: true
I20251025 20:07:58.880088 20131 leader_election.cc:304] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5ed23454b42e88f8828edd920c14e, 116ab0ab9b04415fba8a95256498bc33; no voters:
I20251025 20:07:58.880141 20056 raft_consensus.cc:2468] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f5ed23454b42e88f8828edd920c14e in term 0.
I20251025 20:07:58.880304 20417 raft_consensus.cc:2804] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:07:58.880381 20417 raft_consensus.cc:493] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:07:58.880441 20417 raft_consensus.cc:3060] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.881349 20417 raft_consensus.cc:515] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.881669 20417 leader_election.cc:290] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [CANDIDATE]: Term 1 election: Requested vote from peers e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65:37235), 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563)
I20251025 20:07:58.881974 20056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8db67fc1c1745caa4ad40cefd3009f7" candidate_uuid: "04f5ed23454b42e88f8828edd920c14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f60abded3b4edda9c178d35c1b12dd"
I20251025 20:07:58.882033 20340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8db67fc1c1745caa4ad40cefd3009f7" candidate_uuid: "04f5ed23454b42e88f8828edd920c14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ab0ab9b04415fba8a95256498bc33"
I20251025 20:07:58.882102 20056 raft_consensus.cc:3060] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.882154 20340 raft_consensus.cc:3060] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:07:58.882934 20056 raft_consensus.cc:2468] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f5ed23454b42e88f8828edd920c14e in term 1.
I20251025 20:07:58.883086 20340 raft_consensus.cc:2468] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f5ed23454b42e88f8828edd920c14e in term 1.
I20251025 20:07:58.883371 20131 leader_election.cc:304] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f5ed23454b42e88f8828edd920c14e, e5f60abded3b4edda9c178d35c1b12dd; no voters:
I20251025 20:07:58.883545 20417 raft_consensus.cc:2804] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:07:58.883735 20417 raft_consensus.cc:697] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [term 1 LEADER]: Becoming Leader. State: Replica: 04f5ed23454b42e88f8828edd920c14e, State: Running, Role: LEADER
I20251025 20:07:58.883942 20417 consensus_queue.cc:237] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } }
I20251025 20:07:58.884974 20451 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: TxnStatusTablet state changed. Reason: New leader 04f5ed23454b42e88f8828edd920c14e. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.885077 20451 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:07:58.885325 20453 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 20:07:58.885481 20453 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 20:07:58.886049 19895 catalog_manager.cc:5649] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 04f5ed23454b42e88f8828edd920c14e (127.18.82.66). New cstate: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20251025 20:07:58.912535 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.909384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:41191 (local address 127.0.0.1:53916)
1025 20:07:58.909561 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.909577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.909679 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.909967 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.909969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.909975 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:07:58.910140 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.910144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.911047 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.911049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.911422 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.911427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.911503 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.911538 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.912338 (+ 800us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.912346 (+ 8us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.912380 (+ 34us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.912433 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251025 20:07:58.912570 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.909480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.18.82.66:41191)
1025 20:07:58.909652 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.909657 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.909672 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:07:58.909800 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.909804 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.909866 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.909955 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:07:58.910226 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.910935 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.911508 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.911683 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.911735 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.912240 (+ 505us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.912374 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.912405 (+ 31us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.912452 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20251025 20:07:58.915279 20340 raft_consensus.cc:1275] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Refusing update from remote peer 04f5ed23454b42e88f8828edd920c14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.915503 20056 raft_consensus.cc:1275] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Refusing update from remote peer 04f5ed23454b42e88f8828edd920c14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.915740 20451 consensus_queue.cc:1048] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.915956 20417 consensus_queue.cc:1048] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.917234 20415 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: TxnStatusTablet state changed. Reason: New leader 04f5ed23454b42e88f8828edd920c14e. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.917330 20415 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.917590 20428 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: TxnStatusTablet state changed. Reason: New leader 04f5ed23454b42e88f8828edd920c14e. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.917690 20428 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.920040 20458 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.920073 20417 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.920178 20417 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:07:58.920295 20458 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 04f5ed23454b42e88f8828edd920c14e: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:07:58.921289 20428 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.921375 20428 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P e5f60abded3b4edda9c178d35c1b12dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.922098 20415 tablet_replica.cc:442] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04f5ed23454b42e88f8828edd920c14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f60abded3b4edda9c178d35c1b12dd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 37235 } } peers { permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 } } peers { permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 } } }
I20251025 20:07:58.922185 20415 tablet_replica.cc:445] T a8db67fc1c1745caa4ad40cefd3009f7 P 116ab0ab9b04415fba8a95256498bc33: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:07:58.932708 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.930015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:50623 (local address 127.18.82.66:41191)
1025 20:07:58.930147 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.930151 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.930306 (+ 155us) server_negotiation.cc:408] Connection header received
1025 20:07:58.930429 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.930431 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.930484 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.930542 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:07:58.930934 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.931353 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.932169 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.932307 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.932340 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.932414 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.932466 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.932566 (+ 100us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.932600 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:07:58.932776 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.929938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:41191 (local address 127.18.82.65:50623)
1025 20:07:58.930221 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.930237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.930343 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.930561 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.930564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.930573 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:07:58.930837 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.930845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.931438 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.931442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.932075 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.932084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.932192 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.932236 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.932488 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.932495 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.932557 (+ 62us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.932645 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20251025 20:07:58.938853 20198 raft_consensus.cc:1275] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Refusing update from remote peer e5f60abded3b4edda9c178d35c1b12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.938915 20340 raft_consensus.cc:1275] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Refusing update from remote peer e5f60abded3b4edda9c178d35c1b12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:07:58.939312 20428 consensus_queue.cc:1048] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "116ab0ab9b04415fba8a95256498bc33" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.939447 20420 consensus_queue.cc:1048] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f5ed23454b42e88f8828edd920c14e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:07:58.957221 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.954285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:37235 (local address 127.18.82.66:46157)
1025 20:07:58.954556 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1025 20:07:58.954567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:07:58.954664 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:07:58.954866 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:07:58.954869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:07:58.954876 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:07:58.955137 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.955147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.956045 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.956048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:07:58.956513 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:07:58.956522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.956607 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.956653 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.956996 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.956999 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:07:58.957041 (+ 42us) client_negotiation.cc:240] Negotiation successful
1025 20:07:58.957090 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20251025 20:07:58.957244 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.954333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:46157 (local address 127.18.82.65:37235)
1025 20:07:58.954441 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.954445 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.954632 (+ 187us) server_negotiation.cc:408] Connection header received
1025 20:07:58.954738 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.954741 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.954782 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.954857 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:07:58.955266 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.955949 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.956614 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.956804 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.956849 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:07:58.956920 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:07:58.956991 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.957064 (+ 73us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.957110 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251025 20:07:58.981468 19895 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:07:58.981642 19895 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57568:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:07:58.983523 19895 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7eeb5075b91c412c87ee80bfcd4baca5: Sending DeleteTablet for 3 replicas of tablet f5da69042d85418b919aba70c90b618e
I20251025 20:07:58.984004 20036 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da69042d85418b919aba70c90b618e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:56538
I20251025 20:07:58.984195 20433 tablet_replica.cc:333] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: stopping tablet replica
I20251025 20:07:58.984308 20433 raft_consensus.cc:2243] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:07:58.984344 20178 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da69042d85418b919aba70c90b618e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:53904
I20251025 20:07:58.984505 20433 raft_consensus.cc:2272] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:07:58.984517 20435 tablet_replica.cc:333] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: stopping tablet replica
I20251025 20:07:58.984546 20320 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da69042d85418b919aba70c90b618e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:07:58 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:46482
I20251025 20:07:58.984630 20435 raft_consensus.cc:2243] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:07:58.984724 20435 raft_consensus.cc:2272] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:07:58.984726 20434 tablet_replica.cc:333] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: stopping tablet replica
I20251025 20:07:58.984835 20434 raft_consensus.cc:2243] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:07:58.984938 20434 raft_consensus.cc:2272] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:07:58.985263 20433 ts_tablet_manager.cc:1916] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.985540 20435 ts_tablet_manager.cc:1916] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.986028 20434 ts_tablet_manager.cc:1916] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:07:58.987996 20435 ts_tablet_manager.cc:1929] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:07:58.988041 20433 ts_tablet_manager.cc:1929] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:07:58.988063 20435 log.cc:1199] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/f5da69042d85418b919aba70c90b618e
I20251025 20:07:58.988101 20433 log.cc:1199] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/f5da69042d85418b919aba70c90b618e
I20251025 20:07:58.988329 20435 ts_tablet_manager.cc:1950] T f5da69042d85418b919aba70c90b618e P 04f5ed23454b42e88f8828edd920c14e: Deleting consensus metadata
I20251025 20:07:58.988370 20433 ts_tablet_manager.cc:1950] T f5da69042d85418b919aba70c90b618e P e5f60abded3b4edda9c178d35c1b12dd: Deleting consensus metadata
I20251025 20:07:58.988484 20434 ts_tablet_manager.cc:1929] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:07:58.988559 20434 log.cc:1199] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/f5da69042d85418b919aba70c90b618e
I20251025 20:07:58.988822 20434 ts_tablet_manager.cc:1950] T f5da69042d85418b919aba70c90b618e P 116ab0ab9b04415fba8a95256498bc33: Deleting consensus metadata
I20251025 20:07:58.989025 19882 catalog_manager.cc:4985] TS 04f5ed23454b42e88f8828edd920c14e (127.18.82.66:41191): tablet f5da69042d85418b919aba70c90b618e (table test-table [id=b6705afcc9af4406b7a89ed103e34043]) successfully deleted
I20251025 20:07:58.989025 19883 catalog_manager.cc:4985] TS e5f60abded3b4edda9c178d35c1b12dd (127.18.82.65:37235): tablet f5da69042d85418b919aba70c90b618e (table test-table [id=b6705afcc9af4406b7a89ed103e34043]) successfully deleted
I20251025 20:07:58.989414 19883 catalog_manager.cc:4985] TS 116ab0ab9b04415fba8a95256498bc33 (127.18.82.67:34563): tablet f5da69042d85418b919aba70c90b618e (table test-table [id=b6705afcc9af4406b7a89ed103e34043]) successfully deleted
Oct 25 20:07:58 dist-test-slave-d2pk krb5kdc[19849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:07:58.998715 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:58.989676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.18.82.65:37235)
1025 20:07:58.989802 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:58.989806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:58.990222 (+ 416us) server_negotiation.cc:408] Connection header received
1025 20:07:58.990344 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:58.990347 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:58.990397 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:58.990455 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:58.991435 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.991934 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:58.992784 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:58.992915 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:58.995343 (+ 2428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:58.995365 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:58.995379 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:58.995407 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:58.997252 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.997687 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.997693 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.997695 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.997759 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:58.997930 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:58.997933 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:58.997935 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:58.998134 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:58.998334 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:58.998427 (+ 93us) server_negotiation.cc:300] Negotiation successful
1025 20:07:58.998560 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20251025 20:07:58.999145 20096 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56550
I20251025 20:07:59.007095 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:07:59.000720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.18.82.126:42439)
1025 20:07:59.000827 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1025 20:07:59.000830 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:07:59.000841 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:07:59.000957 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:07:59.000960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:07:59.000996 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:07:59.001049 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:07:59.001911 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:59.002357 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:07:59.003083 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:07:59.003199 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:07:59.004269 (+ 1070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:07:59.004284 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:07:59.004286 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:07:59.004306 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:07:59.005483 (+ 1177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:59.005905 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:59.005907 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:59.005908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:59.005942 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:07:59.006175 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:07:59.006179 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:07:59.006181 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:07:59.006363 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:07:59.006483 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1025 20:07:59.006800 (+ 317us) server_negotiation.cc:300] Negotiation successful
1025 20:07:59.006911 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20251025 20:07:59.007465 19895 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57584
I20251025 20:07:59.008615 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19971
I20251025 20:07:59.020123 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20113
I20251025 20:07:59.026592 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20255
I20251025 20:07:59.032030 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 19865
2025-10-25T20:07:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3407 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:07:59 dist-test-slave-d2pk krb5kdc[20474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:07:59 dist-test-slave-d2pk krb5kdc[20474](info): set up 2 sockets
Oct 25 20:07:59 dist-test-slave-d2pk krb5kdc[20474](info): commencing operation
krb5kdc: starting...
W20251025 20:08:00.890205 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.836s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:00 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:01.012058 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44027
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:44027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:01.113653 20490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:01.113859 20490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:01.113936 20490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:01.116938 20490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:01.116990 20490 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 20:08:01.117017 20490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:01.117053 20490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:01.117089 20490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:01.117127 20490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:01.121058 20490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:44027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:01.121973 20490 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:01.122737 20490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:01.127776 20495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.127776 20496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.127856 20498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:01.127990 20490 server_base.cc:1047] running on GCE node
I20251025 20:08:01.128515 20490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:01.129303 20490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:01.130445 20490 hybrid_clock.cc:648] HybridClock initialized: now 1761422881130414 us; error 43 us; skew 500 ppm
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:01.132767 20490 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:01.133687 20490 webserver.cc:492] Webserver started at http://127.18.82.126:34303/ using document root <none> and password file <none>
I20251025 20:08:01.134152 20490 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:01.134202 20490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:01.134362 20490 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:01.135795 20490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "a68574ac58bb46b68f33c9bef4783bb6"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "3e51e965802142f647c837fb0412734d"
server_key_iv: "14c1d4e4d19f22a0de88afe0e6224843"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.136212 20490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "a68574ac58bb46b68f33c9bef4783bb6"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "3e51e965802142f647c837fb0412734d"
server_key_iv: "14c1d4e4d19f22a0de88afe0e6224843"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.139357 20490 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:01.141229 20505 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.142134 20490 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:01.142233 20490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "a68574ac58bb46b68f33c9bef4783bb6"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "3e51e965802142f647c837fb0412734d"
server_key_iv: "14c1d4e4d19f22a0de88afe0e6224843"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.142308 20490 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:01.158005 20490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:01.160645 20490 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:01.160804 20490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:01.168053 20490 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:44027
I20251025 20:08:01.168056 20567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:44027 every 8 connection(s)
I20251025 20:08:01.168983 20490 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:01.171653 20568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:01.176175 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Bootstrap starting.
I20251025 20:08:01.177271 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20490
I20251025 20:08:01.177361 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:01.177651 18761 external_mini_cluster.cc:1468] Setting key 147bc34faa0b68dc6de21dd12e385967
I20251025 20:08:01.178524 20568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:01.179281 20568 log.cc:826] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:01.180948 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: No bootstrap required, opened a new log
I20251025 20:08:01.183437 20568 raft_consensus.cc:359] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } }
I20251025 20:08:01.183605 20568 raft_consensus.cc:385] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:01.183677 20568 raft_consensus.cc:740] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68574ac58bb46b68f33c9bef4783bb6, State: Initialized, Role: FOLLOWER
I20251025 20:08:01.184147 20568 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } }
I20251025 20:08:01.184303 20568 raft_consensus.cc:399] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:01.184376 20568 raft_consensus.cc:493] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:01.184468 20568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:01.185889 20568 raft_consensus.cc:515] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } }
I20251025 20:08:01.186226 20568 leader_election.cc:304] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68574ac58bb46b68f33c9bef4783bb6; no voters:
I20251025 20:08:01.186532 20568 leader_election.cc:290] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:01.186650 20573 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:01.186851 20573 raft_consensus.cc:697] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [term 1 LEADER]: Becoming Leader. State: Replica: a68574ac58bb46b68f33c9bef4783bb6, State: Running, Role: LEADER
I20251025 20:08:01.187155 20573 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } }
I20251025 20:08:01.187585 20568 sys_catalog.cc:565] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:01.188333 20573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68574ac58bb46b68f33c9bef4783bb6. Latest consensus state: current_term: 1 leader_uuid: "a68574ac58bb46b68f33c9bef4783bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } } }
I20251025 20:08:01.188464 20573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:01.188855 20575 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68574ac58bb46b68f33c9bef4783bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68574ac58bb46b68f33c9bef4783bb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44027 } } }
I20251025 20:08:01.189008 20575 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:01.189617 20581 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:01.190269 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.178619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.18.82.126:44027)
1025 20:08:01.179031 (+ 412us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.179041 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.179072 (+ 31us) server_negotiation.cc:408] Connection header received
1025 20:08:01.179585 (+ 513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.179611 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.179954 (+ 343us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.180251 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.181098 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.181853 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.182452 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.182719 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.185016 (+ 2297us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.185034 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.185043 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.185073 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.187158 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.187655 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.187660 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.187665 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.187754 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.188035 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.188038 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.188040 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.188465 (+ 425us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.189148 (+ 683us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.189269 (+ 121us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.189550 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20251025 20:08:01.193238 20581 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:01.198472 20581 catalog_manager.cc:1357] Generated new cluster ID: f723aec4342a4c4799bd17ccee20c632
I20251025 20:08:01.198534 20581 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:01.212106 20581 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:01.213230 20581 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:01.224290 20581 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Generated new TSK 0
I20251025 20:08:01.225061 20581 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:01.276762 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44027
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:01.381192 20596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:01.381379 20596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:01.381470 20596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:01.384258 20596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:01.384310 20596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:01.384389 20596 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 20:08:01.384415 20596 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:01.384476 20596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:01.388231 20596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:01.389197 20596 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:01.390105 20596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:01.395682 20604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.395707 20601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.395694 20602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:01.395921 20596 server_base.cc:1047] running on GCE node
I20251025 20:08:01.396267 20596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:01.396768 20596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:01.397917 20596 hybrid_clock.cc:648] HybridClock initialized: now 1761422881397896 us; error 32 us; skew 500 ppm
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:01.400218 20596 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:01.401149 20596 webserver.cc:492] Webserver started at http://127.18.82.65:44617/ using document root <none> and password file <none>
I20251025 20:08:01.401698 20596 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:01.401753 20596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:01.401925 20596 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:01.403443 20596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "219e0529732b42829f9789d9518a460d"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "9a200f07790f87ee1c65ce74280daad6"
server_key_iv: "bc078ea3d0260f20caf25ba1c64e85ee"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.403919 20596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "219e0529732b42829f9789d9518a460d"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "9a200f07790f87ee1c65ce74280daad6"
server_key_iv: "bc078ea3d0260f20caf25ba1c64e85ee"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.407011 20596 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251025 20:08:01.409003 20611 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.409941 20596 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:01.410034 20596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "219e0529732b42829f9789d9518a460d"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "9a200f07790f87ee1c65ce74280daad6"
server_key_iv: "bc078ea3d0260f20caf25ba1c64e85ee"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.410111 20596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:01.426036 20596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:01.428423 20596 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:01.428581 20596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:01.429972 20596 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:01.430022 20596 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.430078 20596 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:01.430104 20596 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:01.441059 20596 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:43053
I20251025 20:08:01.441124 20726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:43053 every 8 connection(s)
I20251025 20:08:01.441694 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.430534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:47799 (local address 127.18.82.126:44027)
1025 20:08:01.430684 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.430690 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.431393 (+ 703us) server_negotiation.cc:408] Connection header received
1025 20:08:01.432443 (+ 1050us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.432449 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.432509 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.432595 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.434278 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.434829 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.435451 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.435637 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.438050 (+ 2413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.438069 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.438072 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.438096 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.439706 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.440222 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.440225 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.440226 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.440269 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.440534 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.440536 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.440537 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.440723 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.440811 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.441393 (+ 582us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.441547 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 20:08:01.442059 20596 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:01.442207 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.430756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.65:47799)
1025 20:08:01.431214 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.431263 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.432182 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.432779 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.432791 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.433326 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.434117 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.434133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.434959 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.434962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.435337 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.435344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.435552 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.436164 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.436193 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.437875 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.439811 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.439819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.439829 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.440137 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.440351 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.440354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.440356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.440456 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.440863 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.440870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.441113 (+ 243us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.441324 (+ 211us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.441543 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
I20251025 20:08:01.442243 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20596
I20251025 20:08:01.442353 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:01.442605 18761 external_mini_cluster.cc:1468] Setting key b00a252d5325adc4364fe45e022780fc
I20251025 20:08:01.443962 20727 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44027
I20251025 20:08:01.444111 20727 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:01.444646 20727 heartbeater.cc:507] Master 127.18.82.126:44027 requested a full tablet report, sending...
I20251025 20:08:01.446188 20522 ts_manager.cc:194] Registered new tserver with Master: 219e0529732b42829f9789d9518a460d (127.18.82.65:43053)
I20251025 20:08:01.447475 20522 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:47799
I20251025 20:08:01.451967 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.445376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.65:34369)
1025 20:08:01.445791 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.445805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.445890 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.446117 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.446120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.446273 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.446528 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.446535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.447448 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.447452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.447974 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.447986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.448101 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.448609 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.448625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.448969 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.450866 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.450868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.450870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.451018 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.451234 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.451236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.451237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.451269 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.451632 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.451634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.451705 (+ 71us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.451774 (+ 69us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.451857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":92,"threads_started":1}
I20251025 20:08:01.452139 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.445754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:34369 (local address 127.18.82.126:44027)
1025 20:08:01.445858 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.445861 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.445872 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:01.445970 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.445973 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.446005 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.446077 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.446642 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.447323 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.448091 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.448298 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.449104 (+ 806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.449124 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.449130 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.449161 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.450765 (+ 1604us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.451105 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.451110 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.451112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.451157 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.451340 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.451344 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.451346 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.451504 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.451659 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.451845 (+ 186us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.451952 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:01.485949 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44027
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:01.585264 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:01.585466 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:01.585561 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:01.588637 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:01.588685 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:01.588737 20738 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 20:08:01.588757 20738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:01.588807 20738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:01.592518 20738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:01.593353 20738 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:01.594120 20738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:01.599887 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.599895 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.599900 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:01.600086 20738 server_base.cc:1047] running on GCE node
I20251025 20:08:01.600425 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:01.600858 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:01.601989 20738 hybrid_clock.cc:648] HybridClock initialized: now 1761422881601975 us; error 26 us; skew 500 ppm
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:01.604375 20738 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:01.605315 20738 webserver.cc:492] Webserver started at http://127.18.82.66:33193/ using document root <none> and password file <none>
I20251025 20:08:01.605837 20738 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:01.605885 20738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:01.606029 20738 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:01.607631 20738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "ca3676d39c3047de8df44c58f3c9898e"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "c2cb38f27e3056fd127a957398c3a7ef"
server_key_iv: "a0aabf17cf13b5a310c92bc858dc56fe"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.608064 20738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "ca3676d39c3047de8df44c58f3c9898e"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "c2cb38f27e3056fd127a957398c3a7ef"
server_key_iv: "a0aabf17cf13b5a310c92bc858dc56fe"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.611167 20738 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:01.613052 20753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.613952 20738 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:01.614033 20738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "ca3676d39c3047de8df44c58f3c9898e"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "c2cb38f27e3056fd127a957398c3a7ef"
server_key_iv: "a0aabf17cf13b5a310c92bc858dc56fe"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.614109 20738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:01.628783 20738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:01.631042 20738 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:01.631170 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:01.632375 20738 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:01.632421 20738 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.632457 20738 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:01.632489 20738 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:01.642795 20738 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:40995
I20251025 20:08:01.642820 20868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:40995 every 8 connection(s)
I20251025 20:08:01.643148 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.632983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39589 (local address 127.18.82.126:44027)
1025 20:08:01.633135 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.633140 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.633751 (+ 611us) server_negotiation.cc:408] Connection header received
1025 20:08:01.634541 (+ 790us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.634544 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.634604 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.634680 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.636021 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.636653 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.637261 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.637430 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.639671 (+ 2241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.639688 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.639690 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.639713 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.641143 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.641784 (+ 641us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.641789 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.641792 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.641846 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.642134 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.642136 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.642137 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.642272 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.642335 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.642921 (+ 586us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.643025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:08:01.643729 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.633195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.66:39589)
1025 20:08:01.633622 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.633659 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.634351 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.634844 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.634858 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.635346 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.635878 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.635890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.636762 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.636766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.637148 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.637155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.637339 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.637966 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.638002 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.639520 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.641260 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.641267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.641279 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.641672 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.641943 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.641946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.641948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.642038 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.642362 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.642370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.642637 (+ 267us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.642847 (+ 210us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.643092 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20251025 20:08:01.643739 20738 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:01.645428 20869 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44027
I20251025 20:08:01.645612 20869 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:01.646095 20869 heartbeater.cc:507] Master 127.18.82.126:44027 requested a full tablet report, sending...
I20251025 20:08:01.647075 20522 ts_manager.cc:194] Registered new tserver with Master: ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995)
I20251025 20:08:01.647591 20522 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:39589
I20251025 20:08:01.650769 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20738
I20251025 20:08:01.650888 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:01.651126 18761 external_mini_cluster.cc:1468] Setting key e8e112d8541a7cd73850bf59b2e98dc5
I20251025 20:08:01.653059 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.646846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:44119 (local address 127.18.82.126:44027)
1025 20:08:01.647845 (+ 999us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.647848 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.647860 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:01.647885 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.647887 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.647916 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.647975 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.648485 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.649109 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.649643 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.649756 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.650394 (+ 638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.650407 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.650409 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.650422 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.651686 (+ 1264us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.652010 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.652013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.652015 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.652071 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.652342 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.652347 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.652348 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.652521 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.652620 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.652864 (+ 244us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.652946 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939}
I20251025 20:08:01.653195 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.646775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.66:44119)
1025 20:08:01.647011 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.647023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.647091 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.648028 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.648033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.648180 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.648371 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.648376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.649194 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.649196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.649558 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.649565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.649652 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.650069 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.650081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.650301 (+ 220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.651791 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.651794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.651795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.651934 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.652155 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.652159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.652164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.652219 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.652623 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.652626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.652711 (+ 85us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.652811 (+ 100us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.652932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:01.690332 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44027
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:01.786548 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:01.786746 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:01.786816 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:01.789960 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:01.790010 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:01.790077 20880 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 20:08:01.790105 20880 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:01.790153 20880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:01.793912 20880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:01.794875 20880 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:01.795647 20880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:01.801405 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.801419 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:01.801476 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:01.801729 20880 server_base.cc:1047] running on GCE node
I20251025 20:08:01.802029 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:01.802488 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:01.803619 20880 hybrid_clock.cc:648] HybridClock initialized: now 1761422881803612 us; error 27 us; skew 500 ppm
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:01.805943 20880 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:01.806900 20880 webserver.cc:492] Webserver started at http://127.18.82.67:35357/ using document root <none> and password file <none>
I20251025 20:08:01.807366 20880 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:01.807412 20880 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:01.807560 20880 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:01.809130 20880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "101d6889d19ac8d81ad846438b795f0e"
server_key_iv: "a1ef2c631f5f92aa6dd04585f2dc98d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.809667 20880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "101d6889d19ac8d81ad846438b795f0e"
server_key_iv: "a1ef2c631f5f92aa6dd04585f2dc98d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.812835 20880 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251025 20:08:01.814795 20895 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.815675 20880 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:01.815768 20880 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
format_stamp: "Formatted at 2025-10-25 20:08:01 on dist-test-slave-d2pk"
server_key: "101d6889d19ac8d81ad846438b795f0e"
server_key_iv: "a1ef2c631f5f92aa6dd04585f2dc98d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:01.815829 20880 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:01.833967 20880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:01.836177 20880 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:01.836302 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:01.837680 20880 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:01.837729 20880 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:01.837772 20880 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:01.837807 20880 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:01.848913 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.838162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:49761 (local address 127.18.82.126:44027)
1025 20:08:01.838320 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.838327 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.839073 (+ 746us) server_negotiation.cc:408] Connection header received
1025 20:08:01.840135 (+ 1062us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.840139 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.840185 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.840263 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.841651 (+ 1388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.842125 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.842836 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.843009 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.845363 (+ 2354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.845383 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.845385 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.845410 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.847137 (+ 1727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.847580 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.847584 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.847586 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.847645 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.847895 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.847898 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.847900 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.848072 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.848165 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.848624 (+ 459us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.848744 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:01.848970 20880 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:39469
I20251025 20:08:01.849234 21010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:39469 every 8 connection(s)
I20251025 20:08:01.849489 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.838414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.67:49761)
1025 20:08:01.838898 (+ 484us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.838959 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.839939 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.840425 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.840433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.840901 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.841474 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.841490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.842238 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.842241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.842726 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.842737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.842943 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.843494 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.843528 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.845221 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.847239 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.847244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.847254 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.847484 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.847729 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.847733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.847734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.847810 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.848173 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.848182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.848383 (+ 201us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.848570 (+ 187us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.848822 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20251025 20:08:01.850006 20880 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:01.851202 21011 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44027
I20251025 20:08:01.851349 21011 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:01.851827 21011 heartbeater.cc:507] Master 127.18.82.126:44027 requested a full tablet report, sending...
I20251025 20:08:01.852777 20522 ts_manager.cc:194] Registered new tserver with Master: 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469)
I20251025 20:08:01.853447 20522 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:49761
I20251025 20:08:01.854773 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 20880
I20251025 20:08:01.854909 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:01.855196 18761 external_mini_cluster.cc:1468] Setting key 3a3742a3fbb0e2f230f26c69a1537524
I20251025 20:08:01.857198 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:08:01.858541 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.852583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:36339 (local address 127.18.82.126:44027)
1025 20:08:01.852697 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.852701 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.852904 (+ 203us) server_negotiation.cc:408] Connection header received
1025 20:08:01.852987 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.852989 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.853024 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.853080 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.853656 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.854219 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.854846 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.855006 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.855850 (+ 844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.855864 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.855867 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.855890 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.857228 (+ 1338us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.857669 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.857672 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.857673 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.857711 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.857932 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.857934 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.857935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.858041 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.858104 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.858363 (+ 259us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.858443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:08:01.858731 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.852562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.18.82.67:36339)
1025 20:08:01.852811 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.852826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.852911 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.853111 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.853114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.853291 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:01.853539 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.853548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.854316 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.854318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.854721 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.854731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.854843 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.855434 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:01.855449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:01.855730 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:01.857334 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.857337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.857339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.857563 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.857786 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:01.857789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:01.857791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:01.857851 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:01.858117 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:01.858124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:01.858217 (+ 93us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.858336 (+ 119us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.858475 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 20:08:01 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:01.878288 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.871953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.18.82.126:44027)
1025 20:08:01.872077 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.872081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.872093 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:01.872120 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.872122 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.872151 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.872208 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:01.872874 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.873264 (+ 390us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.873887 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.874076 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.875838 (+ 1762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:01.875853 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:01.875855 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:01.875875 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:01.876929 (+ 1054us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.877250 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.877253 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.877254 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.877290 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:01.877546 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:01.877549 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:01.877551 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:01.877679 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:01.877825 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.878040 (+ 215us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.878143 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:01.880465 20522 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:01.882688 20522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:01.893762 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.890143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:39469 (local address 127.0.0.1:49470)
1025 20:08:01.890439 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.890476 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.890606 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.891350 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.891357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.891380 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:01.891617 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.891624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.892745 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.892750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.893264 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.893271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.893371 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.893431 (+ 60us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.893538 (+ 107us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.893635 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20251025 20:08:01.894279 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.890659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.0.0.1:51588)
1025 20:08:01.891232 (+ 573us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.891248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.891319 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.891944 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.891947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.891968 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:01.892181 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.892187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.893067 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.893073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.893902 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.893912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.894001 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.894034 (+ 33us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.894083 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.894142 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":100,"threads_started":1}
I20251025 20:08:01.894706 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.890293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.18.82.67:39469)
1025 20:08:01.890630 (+ 337us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.890636 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.890892 (+ 256us) server_negotiation.cc:408] Connection header received
1025 20:08:01.890979 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.890984 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.891190 (+ 206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.891741 (+ 551us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:01.891854 (+ 113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.892647 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.893390 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.894142 (+ 752us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.894252 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.894315 (+ 63us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.894567 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
I20251025 20:08:01.895449 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.891180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.18.82.65:43053)
1025 20:08:01.891623 (+ 443us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.891629 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.891646 (+ 17us) server_negotiation.cc:408] Connection header received
1025 20:08:01.891708 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.891711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.891827 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.891922 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:01.892303 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.892945 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.894580 (+ 1635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.895084 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.895167 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.895221 (+ 54us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.895291 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":86,"threads_started":1}
I20251025 20:08:01.896420 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.890778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:40995 (local address 127.0.0.1:40642)
1025 20:08:01.892104 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:01.892113 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:01.892195 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:01.893577 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:01.893581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:01.893599 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:01.893811 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.893817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.895397 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.895400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:01.896093 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:01.896102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.896197 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.896214 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 20:08:01.896256 (+ 42us) client_negotiation.cc:240] Negotiation successful
1025 20:08:01.896302 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"thread_start_us":88,"threads_started":1}
I20251025 20:08:01.896979 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:01.890916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.18.82.66:40995)
1025 20:08:01.891299 (+ 383us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:01.891305 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:01.892196 (+ 891us) server_negotiation.cc:408] Connection header received
1025 20:08:01.893130 (+ 934us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:01.893135 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:01.893273 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:01.893386 (+ 113us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:01.893931 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.895256 (+ 1325us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:01.896215 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:01.896696 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:01.896763 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:01.896809 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 20:08:01.896859 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20251025 20:08:01.897346 20945 tablet_service.cc:1505] Processing CreateTablet for tablet 52425e6a745640c6a7299469291f3848 (DEFAULT_TABLE table=test-table [id=7ddf65971b864ea7962a0db51e5bab5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:01.897861 20661 tablet_service.cc:1505] Processing CreateTablet for tablet 52425e6a745640c6a7299469291f3848 (DEFAULT_TABLE table=test-table [id=7ddf65971b864ea7962a0db51e5bab5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:01.898379 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52425e6a745640c6a7299469291f3848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:01.898743 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52425e6a745640c6a7299469291f3848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:01.899196 20803 tablet_service.cc:1505] Processing CreateTablet for tablet 52425e6a745640c6a7299469291f3848 (DEFAULT_TABLE table=test-table [id=7ddf65971b864ea7962a0db51e5bab5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:01.899823 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52425e6a745640c6a7299469291f3848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:01.904455 21034 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Bootstrap starting.
I20251025 20:08:01.904822 21035 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Bootstrap starting.
I20251025 20:08:01.904888 21036 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Bootstrap starting.
I20251025 20:08:01.907155 21036 tablet_bootstrap.cc:654] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:01.907155 21035 tablet_bootstrap.cc:654] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:01.907335 21034 tablet_bootstrap.cc:654] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:01.908320 21034 log.cc:826] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:01.908418 21036 log.cc:826] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:01.908418 21035 log.cc:826] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:01.909978 21034 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: No bootstrap required, opened a new log
I20251025 20:08:01.910127 21036 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: No bootstrap required, opened a new log
I20251025 20:08:01.910127 21035 tablet_bootstrap.cc:492] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: No bootstrap required, opened a new log
I20251025 20:08:01.910154 21034 ts_tablet_manager.cc:1403] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 20:08:01.910301 21036 ts_tablet_manager.cc:1403] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 20:08:01.910301 21035 ts_tablet_manager.cc:1403] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 20:08:01.912572 21034 raft_consensus.cc:359] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.912766 21034 raft_consensus.cc:385] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:01.912818 21034 raft_consensus.cc:740] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219e0529732b42829f9789d9518a460d, State: Initialized, Role: FOLLOWER
I20251025 20:08:01.912982 21035 raft_consensus.cc:359] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.913180 21035 raft_consensus.cc:385] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:01.913236 21035 raft_consensus.cc:740] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Initialized, Role: FOLLOWER
I20251025 20:08:01.913234 21036 raft_consensus.cc:359] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.913270 21034 consensus_queue.cc:260] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.913434 21036 raft_consensus.cc:385] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:01.913486 21036 raft_consensus.cc:740] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Initialized, Role: FOLLOWER
I20251025 20:08:01.913661 21035 consensus_queue.cc:260] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.913838 21036 consensus_queue.cc:260] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:01.914131 21034 ts_tablet_manager.cc:1434] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20251025 20:08:01.914407 21011 heartbeater.cc:499] Master 127.18.82.126:44027 was elected leader, sending a full tablet report...
I20251025 20:08:01.914541 21035 ts_tablet_manager.cc:1434] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 20:08:01.914587 21036 ts_tablet_manager.cc:1434] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:08:01.914882 20869 heartbeater.cc:499] Master 127.18.82.126:44027 was elected leader, sending a full tablet report...
I20251025 20:08:01.915504 20727 heartbeater.cc:499] Master 127.18.82.126:44027 was elected leader, sending a full tablet report...
W20251025 20:08:01.943457 20728 tablet.cc:2378] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:02.044361 21042 raft_consensus.cc:493] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:02.044543 21042 raft_consensus.cc:515] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:02.045742 21042 leader_election.cc:290] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469), 219e0529732b42829f9789d9518a460d (127.18.82.65:43053)
I20251025 20:08:02.048513 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.045930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:39469 (local address 127.18.82.66:45021)
1025 20:08:02.046094 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.046110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.046208 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.046543 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.046546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.046565 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:02.046783 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.046790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.047541 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.047544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.048130 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.048137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.048249 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.048273 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.048325 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.048384 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20251025 20:08:02.048820 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.045930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.18.82.66:59675)
1025 20:08:02.046523 (+ 593us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.046534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.046649 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.046900 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.046903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.046921 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:02.047158 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.047163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.047882 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.047885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.048439 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.048446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.048556 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.048574 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.048625 (+ 51us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.048684 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":187,"thread_start_us":126,"threads_started":1}
I20251025 20:08:02.049036 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.046102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:45021 (local address 127.18.82.67:39469)
1025 20:08:02.046243 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.046249 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.046264 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:02.046317 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.046330 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.046409 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.046506 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:02.046910 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.047425 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.048259 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.048735 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.048781 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.048839 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.048902 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:02.049285 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.046061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:59675 (local address 127.18.82.65:43053)
1025 20:08:02.046208 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.046213 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.046618 (+ 405us) server_negotiation.cc:408] Connection header received
1025 20:08:02.046762 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.046766 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.046812 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.046879 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:02.047276 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.047764 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.048563 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.049070 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.049111 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.049163 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.049206 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:08:02.049605 20964 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52425e6a745640c6a7299469291f3848" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" is_pre_election: true
I20251025 20:08:02.049837 20964 raft_consensus.cc:2468] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 0.
I20251025 20:08:02.049827 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52425e6a745640c6a7299469291f3848" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d" is_pre_election: true
I20251025 20:08:02.050043 20681 raft_consensus.cc:2468] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 0.
I20251025 20:08:02.050254 20754 leader_election.cc:304] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c9cf35e1541bbbb8a43a106f77448, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:02.050446 21042 raft_consensus.cc:2804] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:02.050513 21042 raft_consensus.cc:493] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:02.050549 21042 raft_consensus.cc:3060] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.051396 21042 raft_consensus.cc:515] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:02.051708 21042 leader_election.cc:290] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 election: Requested vote from peers 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469), 219e0529732b42829f9789d9518a460d (127.18.82.65:43053)
I20251025 20:08:02.052009 20964 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52425e6a745640c6a7299469291f3848" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
I20251025 20:08:02.052031 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52425e6a745640c6a7299469291f3848" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d"
I20251025 20:08:02.052117 20964 raft_consensus.cc:3060] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.052127 20681 raft_consensus.cc:3060] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.052956 20964 raft_consensus.cc:2468] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 1.
I20251025 20:08:02.052973 20681 raft_consensus.cc:2468] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 1.
I20251025 20:08:02.053224 20754 leader_election.cc:304] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c9cf35e1541bbbb8a43a106f77448, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:02.053361 21042 raft_consensus.cc:2804] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:02.053589 21042 raft_consensus.cc:697] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 1 LEADER]: Becoming Leader. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Running, Role: LEADER
I20251025 20:08:02.053897 21042 consensus_queue.cc:237] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } }
I20251025 20:08:02.056875 20519 catalog_manager.cc:5649] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3676d39c3047de8df44c58f3c9898e (127.18.82.66). New cstate: current_term: 1 leader_uuid: "ca3676d39c3047de8df44c58f3c9898e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } health_report { overall_health: UNKNOWN } } }
W20251025 20:08:02.100785 21012 tablet.cc:2378] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 25 20:08:02 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:02.105823 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.099361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44027 (local address 127.0.0.1:36838)
1025 20:08:02.099667 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.099680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.099795 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.100003 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.100006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.100441 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:02.100625 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.100632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.101307 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.101312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.101695 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.101703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.101815 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.102331 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:02.102343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:02.103545 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:02.104807 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:02.104814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:02.104818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:02.104995 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:02.105151 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:02.105153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:02.105154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:02.105190 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:02.105408 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:02.105413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:02.105499 (+ 86us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.105606 (+ 107us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.105709 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":111,"threads_started":1}
I20251025 20:08:02.105856 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.099525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.18.82.126:44027)
1025 20:08:02.099656 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.099661 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.099750 (+ 89us) server_negotiation.cc:408] Connection header received
1025 20:08:02.099877 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.099880 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.099916 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.099975 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:02.100736 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.101214 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.101810 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.101963 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.103655 (+ 1692us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:02.103669 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:02.103671 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:02.103692 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:02.104719 (+ 1027us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:02.105064 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:02.105066 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:02.105067 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:02.105097 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:02.105239 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:02.105241 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:02.105242 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:02.105344 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:02.105406 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.105650 (+ 244us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.105760 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":49}
I20251025 20:08:02.109268 20519 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:36838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 20:08:02.109952 20519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:02.114518 20661 tablet_service.cc:1505] Processing CreateTablet for tablet 7284343bd6834fb0964a06f89829c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9d06818cfb4a49dda03ea187423bd32d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:02.114678 20803 tablet_service.cc:1505] Processing CreateTablet for tablet 7284343bd6834fb0964a06f89829c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9d06818cfb4a49dda03ea187423bd32d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:02.114820 20945 tablet_service.cc:1505] Processing CreateTablet for tablet 7284343bd6834fb0964a06f89829c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9d06818cfb4a49dda03ea187423bd32d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:02.115190 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7284343bd6834fb0964a06f89829c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.115276 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7284343bd6834fb0964a06f89829c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.115288 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7284343bd6834fb0964a06f89829c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.117491 21036 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: Bootstrap starting.
I20251025 20:08:02.117959 21034 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: Bootstrap starting.
I20251025 20:08:02.118347 21035 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: Bootstrap starting.
I20251025 20:08:02.118636 21036 tablet_bootstrap.cc:654] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.119026 21034 tablet_bootstrap.cc:654] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.119351 21035 tablet_bootstrap.cc:654] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.120301 21036 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: No bootstrap required, opened a new log
I20251025 20:08:02.120385 21036 ts_tablet_manager.cc:1403] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251025 20:08:02.120678 21035 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: No bootstrap required, opened a new log
I20251025 20:08:02.120730 21034 tablet_bootstrap.cc:492] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: No bootstrap required, opened a new log
I20251025 20:08:02.120759 21035 ts_tablet_manager.cc:1403] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:02.120810 21034 ts_tablet_manager.cc:1403] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251025 20:08:02.120890 21036 raft_consensus.cc:359] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121001 21036 raft_consensus.cc:385] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.121035 21036 raft_consensus.cc:740] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.121152 21036 consensus_queue.cc:260] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121201 21035 raft_consensus.cc:359] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121260 21034 raft_consensus.cc:359] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121310 21035 raft_consensus.cc:385] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.121340 21034 raft_consensus.cc:385] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.121346 21035 raft_consensus.cc:740] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.121369 21034 raft_consensus.cc:740] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219e0529732b42829f9789d9518a460d, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.121423 21036 ts_tablet_manager.cc:1434] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 20:08:02.121408 21042 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.121459 21035 consensus_queue.cc:260] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121491 21034 consensus_queue.cc:260] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.121558 21042 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.121748 21034 ts_tablet_manager.cc:1434] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:02.121766 21035 ts_tablet_manager.cc:1434] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:02.121968 21041 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.122068 21041 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.122308 21040 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.122424 21040 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.126497 21040 raft_consensus.cc:493] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:02.126576 21040 raft_consensus.cc:515] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.127488 21040 leader_election.cc:290] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995), 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469)
I20251025 20:08:02.130285 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.127709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:40995 (local address 127.18.82.65:38531)
1025 20:08:02.127927 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.127943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.128049 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.128272 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.128275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.128294 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:02.128600 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.128610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.129292 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.129295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.129940 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.129948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.130047 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.130068 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.130113 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.130163 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20251025 20:08:02.130585 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.128008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:39469 (local address 127.18.82.65:49941)
1025 20:08:02.128437 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.128450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.128535 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.128754 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.128758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.128789 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:02.128986 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.128992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.129667 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.129669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.130141 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.130148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.130244 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.130261 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.130306 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.130466 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":63,"threads_started":1}
I20251025 20:08:02.130760 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.127801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:38531 (local address 127.18.82.66:40995)
1025 20:08:02.127928 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.127934 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.128011 (+ 77us) server_negotiation.cc:408] Connection header received
1025 20:08:02.128141 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.128144 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.128184 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.128243 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:02.128706 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.129193 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.130049 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.130476 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.130512 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.130570 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.130648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:02.130910 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.128256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:49941 (local address 127.18.82.67:39469)
1025 20:08:02.128369 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.128372 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.128512 (+ 140us) server_negotiation.cc:408] Connection header received
1025 20:08:02.128609 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.128611 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.128654 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.128728 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:02.129088 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.129522 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.130243 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.130683 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.130720 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.130761 (+ 41us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.130797 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:08:02.131245 20964 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7284343bd6834fb0964a06f89829c1b7" candidate_uuid: "219e0529732b42829f9789d9518a460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" is_pre_election: true
I20251025 20:08:02.131260 20823 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7284343bd6834fb0964a06f89829c1b7" candidate_uuid: "219e0529732b42829f9789d9518a460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e" is_pre_election: true
I20251025 20:08:02.131354 20964 raft_consensus.cc:2468] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219e0529732b42829f9789d9518a460d in term 0.
I20251025 20:08:02.131455 20823 raft_consensus.cc:2468] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219e0529732b42829f9789d9518a460d in term 0.
I20251025 20:08:02.132078 20612 leader_election.cc:304] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219e0529732b42829f9789d9518a460d, 2a3c9cf35e1541bbbb8a43a106f77448; no voters:
I20251025 20:08:02.132279 21040 raft_consensus.cc:2804] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:02.132341 21040 raft_consensus.cc:493] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:02.132386 21040 raft_consensus.cc:3060] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.133118 21040 raft_consensus.cc:515] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.133409 21040 leader_election.cc:290] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [CANDIDATE]: Term 1 election: Requested vote from peers ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995), 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469)
I20251025 20:08:02.133721 20823 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7284343bd6834fb0964a06f89829c1b7" candidate_uuid: "219e0529732b42829f9789d9518a460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e"
I20251025 20:08:02.133766 20964 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7284343bd6834fb0964a06f89829c1b7" candidate_uuid: "219e0529732b42829f9789d9518a460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
I20251025 20:08:02.133836 20964 raft_consensus.cc:3060] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.133852 20823 raft_consensus.cc:3060] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.134538 20964 raft_consensus.cc:2468] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219e0529732b42829f9789d9518a460d in term 1.
I20251025 20:08:02.134593 20823 raft_consensus.cc:2468] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219e0529732b42829f9789d9518a460d in term 1.
I20251025 20:08:02.134796 20612 leader_election.cc:304] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219e0529732b42829f9789d9518a460d, 2a3c9cf35e1541bbbb8a43a106f77448; no voters:
I20251025 20:08:02.134948 21040 raft_consensus.cc:2804] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:02.135169 21040 raft_consensus.cc:697] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [term 1 LEADER]: Becoming Leader. State: Replica: 219e0529732b42829f9789d9518a460d, State: Running, Role: LEADER
I20251025 20:08:02.135422 21040 consensus_queue.cc:237] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.137173 21057 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: TxnStatusTablet state changed. Reason: New leader 219e0529732b42829f9789d9518a460d. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.137279 21057 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:02.137562 21059 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 20:08:02.137754 21059 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 20:08:02.138185 20519 catalog_manager.cc:5649] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d reported cstate change: term changed from 0 to 1, leader changed from <none> to 219e0529732b42829f9789d9518a460d (127.18.82.65). New cstate: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } health_report { overall_health: UNKNOWN } } }
W20251025 20:08:02.145170 20870 tablet.cc:2378] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:02.152534 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.149362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.0.0.1:51604)
1025 20:08:02.149596 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.149609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.149702 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.149973 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.149975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.149981 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:02.150162 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.150166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.151021 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.151023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.151369 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.151374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.151445 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.151478 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.152282 (+ 804us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.152293 (+ 11us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.152360 (+ 67us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.152431 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20251025 20:08:02.152575 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.149472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.18.82.65:43053)
1025 20:08:02.149603 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.149608 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.149650 (+ 42us) server_negotiation.cc:408] Connection header received
1025 20:08:02.149817 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.149820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.149876 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.149964 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:02.150269 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.150934 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.151460 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.151647 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.151713 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.152199 (+ 486us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.152297 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.152405 (+ 108us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.152453 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 20:08:02.157091 20823 raft_consensus.cc:1275] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Refusing update from remote peer 219e0529732b42829f9789d9518a460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.157205 20964 raft_consensus.cc:1275] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Refusing update from remote peer 219e0529732b42829f9789d9518a460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.157677 21057 consensus_queue.cc:1048] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:02.157860 21040 consensus_queue.cc:1048] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:02.160665 21042 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: TxnStatusTablet state changed. Reason: New leader 219e0529732b42829f9789d9518a460d. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.160758 21042 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.162763 21040 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.162916 21040 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:02.163113 21062 mvcc.cc:204] Tried to move back new op lower bound from 7214788125309751296 to 7214788125228978176. Current Snapshot: MvccSnapshot[applied={T|T < 7214788125309751296}]
I20251025 20:08:02.163712 21041 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: TxnStatusTablet state changed. Reason: New leader 219e0529732b42829f9789d9518a460d. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.163800 21041 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.163969 21042 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.164053 21042 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P ca3676d39c3047de8df44c58f3c9898e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.165292 21063 mvcc.cc:204] Tried to move back new op lower bound from 7214788125309751296 to 7214788125228978176. Current Snapshot: MvccSnapshot[applied={T|T < 7214788125309751296}]
I20251025 20:08:02.168303 21057 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.168411 21057 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 219e0529732b42829f9789d9518a460d: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:02.172720 21041 tablet_replica.cc:442] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "219e0529732b42829f9789d9518a460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } } }
I20251025 20:08:02.172802 21041 tablet_replica.cc:445] T 7284343bd6834fb0964a06f89829c1b7 P 2a3c9cf35e1541bbbb8a43a106f77448: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:02.174010 21066 mvcc.cc:204] Tried to move back new op lower bound from 7214788125309751296 to 7214788125228978176. Current Snapshot: MvccSnapshot[applied={T|T < 7214788125309751296}]
I20251025 20:08:02.177574 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.174772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.18.82.66:40995)
1025 20:08:02.174953 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.174957 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.174968 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:02.175030 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.175032 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.175093 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.175169 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:02.175577 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.176067 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.176647 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.176808 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.176856 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.177199 (+ 343us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.177291 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.177380 (+ 89us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.177432 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20251025 20:08:02.188349 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.184128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:58247 (local address 127.18.82.65:43053)
1025 20:08:02.184284 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.184288 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.185797 (+ 1509us) server_negotiation.cc:408] Connection header received
1025 20:08:02.185905 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.185908 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.185950 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.186013 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:02.186511 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.186961 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.187831 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.187958 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.187996 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.188055 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.188116 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.188215 (+ 99us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.188252 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251025 20:08:02.188406 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.184052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.18.82.66:58247)
1025 20:08:02.185709 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.185724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.185823 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.186032 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.186035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.186043 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:02.186370 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.186378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.187056 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.187059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.187716 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.187726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.187834 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.187873 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.188136 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.188141 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.188205 (+ 64us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.188270 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":75,"threads_started":1}
I20251025 20:08:02.194687 20964 raft_consensus.cc:1275] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Refusing update from remote peer ca3676d39c3047de8df44c58f3c9898e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.194691 20681 raft_consensus.cc:1275] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Refusing update from remote peer ca3676d39c3047de8df44c58f3c9898e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.195106 21042 consensus_queue.cc:1048] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:02.195254 21044 consensus_queue.cc:1048] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Connected to new peer: Peer: permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:02.215792 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.213019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:60517 (local address 127.18.82.66:40995)
1025 20:08:02.213201 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:02.213205 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:02.213314 (+ 109us) server_negotiation.cc:408] Connection header received
1025 20:08:02.213449 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:02.213452 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:02.213487 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:02.213569 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:02.213975 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.214369 (+ 394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.215168 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.215347 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.215384 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.215439 (+ 55us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.215501 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:02.215629 (+ 128us) server_negotiation.cc:300] Negotiation successful
1025 20:08:02.215675 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20251025 20:08:02.216140 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:02.212965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:40995 (local address 127.18.82.65:60517)
1025 20:08:02.213243 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:02.213257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:02.213358 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:02.213583 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:02.213586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:02.213594 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:02.213826 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.213832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.214488 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:02.214492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:02.215061 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:02.215068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:02.215161 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:02.215202 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:02.215526 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:02.215531 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:08:02.215971 (+ 440us) client_negotiation.cc:240] Negotiation successful
1025 20:08:02.216037 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20251025 20:08:02.238085 20522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:02.238256 20522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:02.240444 20522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Sending DeleteTablet for 3 replicas of tablet 52425e6a745640c6a7299469291f3848
I20251025 20:08:02.241104 20945 tablet_service.cc:1552] Processing DeleteTablet for tablet 52425e6a745640c6a7299469291f3848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:02 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:49470
I20251025 20:08:02.241763 20803 tablet_service.cc:1552] Processing DeleteTablet for tablet 52425e6a745640c6a7299469291f3848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:02 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:40642
I20251025 20:08:02.241755 20522 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:02.242151 20522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:02.242403 21082 tablet_replica.cc:333] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: stopping tablet replica
I20251025 20:08:02.242703 21082 raft_consensus.cc:2243] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:02.243110 21082 raft_consensus.cc:2272] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:02.244381 20661 tablet_service.cc:1552] Processing DeleteTablet for tablet 52425e6a745640c6a7299469291f3848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:02 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:51588
I20251025 20:08:02.244781 21082 ts_tablet_manager.cc:1916] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:02.248273 20945 tablet_service.cc:1505] Processing CreateTablet for tablet 449ad69207cd4b179d880e68f2018f17 (DEFAULT_TABLE table=test-table [id=06097a8aa68742c786d1cc8170bd9a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:02.248838 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ad69207cd4b179d880e68f2018f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.248891 20661 tablet_service.cc:1505] Processing CreateTablet for tablet 449ad69207cd4b179d880e68f2018f17 (DEFAULT_TABLE table=test-table [id=06097a8aa68742c786d1cc8170bd9a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:02.249106 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ad69207cd4b179d880e68f2018f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.249681 21081 tablet_replica.cc:333] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: stopping tablet replica
I20251025 20:08:02.249907 21081 raft_consensus.cc:2243] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:02.250087 21081 raft_consensus.cc:2272] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:02.251142 21035 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Bootstrap starting.
I20251025 20:08:02.251336 21034 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Bootstrap starting.
I20251025 20:08:02.252161 21035 tablet_bootstrap.cc:654] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.252193 21081 ts_tablet_manager.cc:1916] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:02.252243 21034 tablet_bootstrap.cc:654] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.248271 20803 tablet_service.cc:1505] Processing CreateTablet for tablet 449ad69207cd4b179d880e68f2018f17 (DEFAULT_TABLE table=test-table [id=06097a8aa68742c786d1cc8170bd9a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:02.253275 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449ad69207cd4b179d880e68f2018f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:02.254621 21081 ts_tablet_manager.cc:1929] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:02.254676 21034 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: No bootstrap required, opened a new log
I20251025 20:08:02.254716 21081 log.cc:1199] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/52425e6a745640c6a7299469291f3848
I20251025 20:08:02.254762 21034 ts_tablet_manager.cc:1403] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251025 20:08:02.254998 21081 ts_tablet_manager.cc:1950] T 52425e6a745640c6a7299469291f3848 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting consensus metadata
I20251025 20:08:02.255201 21034 raft_consensus.cc:359] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.255321 21034 raft_consensus.cc:385] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.255353 21034 raft_consensus.cc:740] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219e0529732b42829f9789d9518a460d, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.255501 21036 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Bootstrap starting.
I20251025 20:08:02.255462 21034 consensus_queue.cc:260] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.255726 21034 ts_tablet_manager.cc:1434] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:02.248052 21082 ts_tablet_manager.cc:1929] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:02.255990 21082 log.cc:1199] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/52425e6a745640c6a7299469291f3848
I20251025 20:08:02.256362 21082 ts_tablet_manager.cc:1950] T 52425e6a745640c6a7299469291f3848 P ca3676d39c3047de8df44c58f3c9898e: Deleting consensus metadata
I20251025 20:08:02.256403 21036 tablet_bootstrap.cc:654] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:02.257939 20506 catalog_manager.cc:4985] TS ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995): tablet 52425e6a745640c6a7299469291f3848 (table test-table [id=7ddf65971b864ea7962a0db51e5bab5c]) successfully deleted
I20251025 20:08:02.258863 21035 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: No bootstrap required, opened a new log
I20251025 20:08:02.258975 21035 ts_tablet_manager.cc:1403] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251025 20:08:02.259426 21035 raft_consensus.cc:359] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.259522 21035 raft_consensus.cc:385] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.259555 21035 raft_consensus.cc:740] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.259665 21035 consensus_queue.cc:260] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.259922 21035 ts_tablet_manager.cc:1434] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:02.260421 21036 tablet_bootstrap.cc:492] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: No bootstrap required, opened a new log
I20251025 20:08:02.260491 21036 ts_tablet_manager.cc:1403] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251025 20:08:02.260924 21036 raft_consensus.cc:359] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.261039 21036 raft_consensus.cc:385] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:02.261068 21036 raft_consensus.cc:740] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Initialized, Role: FOLLOWER
I20251025 20:08:02.261183 21036 consensus_queue.cc:260] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.261435 21036 ts_tablet_manager.cc:1434] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:02.261823 21083 tablet_replica.cc:333] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: stopping tablet replica
I20251025 20:08:02.262085 21083 raft_consensus.cc:2243] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:02.262283 21083 raft_consensus.cc:2272] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:02.263542 21083 ts_tablet_manager.cc:1916] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:02.265727 21083 ts_tablet_manager.cc:1929] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:02.265796 21083 log.cc:1199] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/52425e6a745640c6a7299469291f3848
I20251025 20:08:02.266211 20506 catalog_manager.cc:4985] TS 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469): tablet 52425e6a745640c6a7299469291f3848 (table test-table [id=7ddf65971b864ea7962a0db51e5bab5c]) successfully deleted
I20251025 20:08:02.266407 21083 ts_tablet_manager.cc:1950] T 52425e6a745640c6a7299469291f3848 P 219e0529732b42829f9789d9518a460d: Deleting consensus metadata
I20251025 20:08:02.267287 20507 catalog_manager.cc:4985] TS 219e0529732b42829f9789d9518a460d (127.18.82.65:43053): tablet 52425e6a745640c6a7299469291f3848 (table test-table [id=7ddf65971b864ea7962a0db51e5bab5c]) successfully deleted
I20251025 20:08:02.465297 21042 raft_consensus.cc:493] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:02.465426 21042 raft_consensus.cc:515] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.465869 21042 leader_election.cc:290] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469)
I20251025 20:08:02.466308 20965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "449ad69207cd4b179d880e68f2018f17" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" is_pre_election: true
I20251025 20:08:02.466291 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "449ad69207cd4b179d880e68f2018f17" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d" is_pre_election: true
I20251025 20:08:02.466488 20965 raft_consensus.cc:2468] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 0.
I20251025 20:08:02.466488 20681 raft_consensus.cc:2468] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 0.
I20251025 20:08:02.466807 20754 leader_election.cc:304] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c9cf35e1541bbbb8a43a106f77448, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:02.466985 21042 raft_consensus.cc:2804] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:02.467043 21042 raft_consensus.cc:493] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:02.467070 21042 raft_consensus.cc:3060] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.467959 21042 raft_consensus.cc:515] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.468271 21042 leader_election.cc:290] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 election: Requested vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469)
I20251025 20:08:02.468595 20965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "449ad69207cd4b179d880e68f2018f17" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3c9cf35e1541bbbb8a43a106f77448"
I20251025 20:08:02.468595 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "449ad69207cd4b179d880e68f2018f17" candidate_uuid: "ca3676d39c3047de8df44c58f3c9898e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d"
I20251025 20:08:02.468719 20965 raft_consensus.cc:3060] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.468747 20681 raft_consensus.cc:3060] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:02.469719 20681 raft_consensus.cc:2468] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 1.
I20251025 20:08:02.469719 20965 raft_consensus.cc:2468] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3676d39c3047de8df44c58f3c9898e in term 1.
I20251025 20:08:02.469982 20755 leader_election.cc:304] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219e0529732b42829f9789d9518a460d, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:02.470155 21042 raft_consensus.cc:2804] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:02.470233 21042 raft_consensus.cc:697] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 1 LEADER]: Becoming Leader. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Running, Role: LEADER
I20251025 20:08:02.470367 21042 consensus_queue.cc:237] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:02.471984 20522 catalog_manager.cc:5649] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3676d39c3047de8df44c58f3c9898e (127.18.82.66). New cstate: current_term: 1 leader_uuid: "ca3676d39c3047de8df44c58f3c9898e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:02.502242 20965 raft_consensus.cc:1275] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Refusing update from remote peer ca3676d39c3047de8df44c58f3c9898e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.502257 20681 raft_consensus.cc:1275] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Refusing update from remote peer ca3676d39c3047de8df44c58f3c9898e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:02.502609 21044 consensus_queue.cc:1048] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:02.502745 21042 consensus_queue.cc:1048] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [LEADER]: Connected to new peer: Peer: permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:08:02.514660 21063 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251025 20:08:04.061537 20521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:04.061700 20521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:04.063845 20521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Sending DeleteTablet for 3 replicas of tablet 449ad69207cd4b179d880e68f2018f17
I20251025 20:08:04.064327 20661 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ad69207cd4b179d880e68f2018f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:51588
I20251025 20:08:04.064361 20803 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ad69207cd4b179d880e68f2018f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:40642
I20251025 20:08:04.064487 20945 tablet_service.cc:1552] Processing DeleteTablet for tablet 449ad69207cd4b179d880e68f2018f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:49470
I20251025 20:08:04.064800 21115 tablet_replica.cc:333] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: stopping tablet replica
I20251025 20:08:04.064977 21115 raft_consensus.cc:2243] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:04.065119 21115 raft_consensus.cc:2272] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.065179 21113 tablet_replica.cc:333] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: stopping tablet replica
I20251025 20:08:04.065377 21113 raft_consensus.cc:2243] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:04.065608 21113 raft_consensus.cc:2272] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.065840 21115 ts_tablet_manager.cc:1916] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.066311 21113 ts_tablet_manager.cc:1916] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.067253 21114 tablet_replica.cc:333] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: stopping tablet replica
I20251025 20:08:04.067396 21114 raft_consensus.cc:2243] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:04.067667 21114 raft_consensus.cc:2272] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.068269 21114 ts_tablet_manager.cc:1916] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.068857 21115 ts_tablet_manager.cc:1929] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 20:08:04.068930 21115 log.cc:1199] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/449ad69207cd4b179d880e68f2018f17
I20251025 20:08:04.069152 21113 ts_tablet_manager.cc:1929] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 20:08:04.069193 21115 ts_tablet_manager.cc:1950] T 449ad69207cd4b179d880e68f2018f17 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting consensus metadata
I20251025 20:08:04.069218 21113 log.cc:1199] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/449ad69207cd4b179d880e68f2018f17
I20251025 20:08:04.069849 20506 catalog_manager.cc:4985] TS 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469): tablet 449ad69207cd4b179d880e68f2018f17 (table test-table [id=06097a8aa68742c786d1cc8170bd9a99]) successfully deleted
I20251025 20:08:04.070632 21114 ts_tablet_manager.cc:1929] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 20:08:04.070711 21114 log.cc:1199] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/449ad69207cd4b179d880e68f2018f17
I20251025 20:08:04.070960 21114 ts_tablet_manager.cc:1950] T 449ad69207cd4b179d880e68f2018f17 P ca3676d39c3047de8df44c58f3c9898e: Deleting consensus metadata
I20251025 20:08:04.071462 20506 catalog_manager.cc:4985] TS ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995): tablet 449ad69207cd4b179d880e68f2018f17 (table test-table [id=06097a8aa68742c786d1cc8170bd9a99]) successfully deleted
I20251025 20:08:04.071702 21113 ts_tablet_manager.cc:1950] T 449ad69207cd4b179d880e68f2018f17 P 219e0529732b42829f9789d9518a460d: Deleting consensus metadata
I20251025 20:08:04.072395 20507 catalog_manager.cc:4985] TS 219e0529732b42829f9789d9518a460d (127.18.82.65:43053): tablet 449ad69207cd4b179d880e68f2018f17 (table test-table [id=06097a8aa68742c786d1cc8170bd9a99]) successfully deleted
Oct 25 20:08:04 dist-test-slave-d2pk krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422884, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 20:08:04 dist-test-slave-d2pk krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422884, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:04.089043 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.082283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.18.82.126:44027)
1025 20:08:04.082514 (+ 231us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.082517 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.082532 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:04.082649 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.082651 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.082690 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.082758 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:04.083400 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.083887 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.084369 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.084496 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.086423 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:04.086440 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:04.086442 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:04.086483 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:04.087837 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:04.088174 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:04.088177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:04.088178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:04.088213 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:04.088356 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:04.088358 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:04.088359 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:04.088482 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:04.088575 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.088718 (+ 143us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.088813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20251025 20:08:04.091171 20521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:04.091568 20521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:04.095681 20661 tablet_service.cc:1505] Processing CreateTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 (DEFAULT_TABLE table=test-table [id=576612433d6543a79ba6a6a3656e2017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.095747 20803 tablet_service.cc:1505] Processing CreateTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 (DEFAULT_TABLE table=test-table [id=576612433d6543a79ba6a6a3656e2017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.095968 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d83cef3ac544c77b1b15a37513cd696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.096001 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d83cef3ac544c77b1b15a37513cd696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.095945 20945 tablet_service.cc:1505] Processing CreateTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 (DEFAULT_TABLE table=test-table [id=576612433d6543a79ba6a6a3656e2017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.096185 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d83cef3ac544c77b1b15a37513cd696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.097723 21124 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Bootstrap starting.
I20251025 20:08:04.098538 21125 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Bootstrap starting.
I20251025 20:08:04.098618 21126 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Bootstrap starting.
I20251025 20:08:04.098765 21124 tablet_bootstrap.cc:654] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.099453 21126 tablet_bootstrap.cc:654] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.099453 21125 tablet_bootstrap.cc:654] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.099797 21124 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: No bootstrap required, opened a new log
I20251025 20:08:04.099879 21124 ts_tablet_manager.cc:1403] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:04.100215 21124 raft_consensus.cc:359] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.100303 21124 raft_consensus.cc:385] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.100337 21124 raft_consensus.cc:740] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219e0529732b42829f9789d9518a460d, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.100466 21125 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: No bootstrap required, opened a new log
I20251025 20:08:04.100535 21125 ts_tablet_manager.cc:1403] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:04.100488 21124 consensus_queue.cc:260] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.100777 21124 ts_tablet_manager.cc:1434] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 20:08:04.100921 21125 raft_consensus.cc:359] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.101006 21125 raft_consensus.cc:385] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.101035 21125 raft_consensus.cc:740] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.101176 21125 consensus_queue.cc:260] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.101527 21125 ts_tablet_manager.cc:1434] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:04.101572 21126 tablet_bootstrap.cc:492] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: No bootstrap required, opened a new log
I20251025 20:08:04.101655 21126 ts_tablet_manager.cc:1403] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:04.102054 21126 raft_consensus.cc:359] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.102154 21126 raft_consensus.cc:385] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.102193 21126 raft_consensus.cc:740] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.102319 21126 consensus_queue.cc:260] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.102553 21126 ts_tablet_manager.cc:1434] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 20:08:04.176568 21130 raft_consensus.cc:493] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:04.176707 21130 raft_consensus.cc:515] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.177706 21130 leader_election.cc:290] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995)
I20251025 20:08:04.180728 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.177954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:40995 (local address 127.18.82.67:46703)
1025 20:08:04.178478 (+ 524us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:04.178493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:04.178603 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:04.178862 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:04.178865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:04.178912 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:04.179139 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.179145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.179826 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.179829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:04.180374 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.180381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.180478 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.180500 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 20:08:04.180545 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:04.180598 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"spinlock_wait_cycles":896,"thread_start_us":122,"threads_started":1}
I20251025 20:08:04.180929 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.177906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.18.82.67:50463)
1025 20:08:04.178409 (+ 503us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:04.178426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:04.178530 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:04.178841 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:04.178844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:04.178861 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:04.179373 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.179380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.180068 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.180070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:04.180585 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.180591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.180684 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.180701 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 20:08:04.180746 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:04.180793 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"spinlock_wait_cycles":14720,"thread_start_us":116,"threads_started":1}
I20251025 20:08:04.181197 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.178060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:46703 (local address 127.18.82.66:40995)
1025 20:08:04.178347 (+ 287us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.178352 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.178585 (+ 233us) server_negotiation.cc:408] Connection header received
1025 20:08:04.178720 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.178722 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.178767 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.178841 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:04.179250 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.179716 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.180482 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.180910 (+ 428us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.180947 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.180998 (+ 51us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.181063 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20251025 20:08:04.181581 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.178290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:50463 (local address 127.18.82.65:43053)
1025 20:08:04.178619 (+ 329us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.178624 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.178647 (+ 23us) server_negotiation.cc:408] Connection header received
1025 20:08:04.178689 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.178692 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.178734 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.178808 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:04.179503 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.179949 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.180694 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.181308 (+ 614us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.181347 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.181393 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.181443 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20251025 20:08:04.181799 20823 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d83cef3ac544c77b1b15a37513cd696" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e" is_pre_election: true
I20251025 20:08:04.181959 20823 raft_consensus.cc:2468] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 0.
I20251025 20:08:04.181973 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d83cef3ac544c77b1b15a37513cd696" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d" is_pre_election: true
I20251025 20:08:04.182107 20681 raft_consensus.cc:2468] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 0.
I20251025 20:08:04.182358 20896 leader_election.cc:304] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c9cf35e1541bbbb8a43a106f77448, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:04.182577 21130 raft_consensus.cc:2804] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:04.182631 21130 raft_consensus.cc:493] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:04.182677 21130 raft_consensus.cc:3060] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.183476 21130 raft_consensus.cc:515] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.183804 21130 leader_election.cc:290] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 election: Requested vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995)
I20251025 20:08:04.184098 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d83cef3ac544c77b1b15a37513cd696" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d"
I20251025 20:08:04.184095 20823 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d83cef3ac544c77b1b15a37513cd696" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e"
I20251025 20:08:04.184185 20681 raft_consensus.cc:3060] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.184238 20823 raft_consensus.cc:3060] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.184949 20681 raft_consensus.cc:2468] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 1.
I20251025 20:08:04.185065 20823 raft_consensus.cc:2468] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 1.
I20251025 20:08:04.185235 20897 leader_election.cc:304] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219e0529732b42829f9789d9518a460d, 2a3c9cf35e1541bbbb8a43a106f77448; no voters:
I20251025 20:08:04.185413 21130 raft_consensus.cc:2804] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:04.185647 21130 raft_consensus.cc:697] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 LEADER]: Becoming Leader. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Running, Role: LEADER
I20251025 20:08:04.185868 21130 consensus_queue.cc:237] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.187975 20522 catalog_manager.cc:5649] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67). New cstate: current_term: 1 leader_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } health_report { overall_health: HEALTHY } } }
I20251025 20:08:04.201130 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.198247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.18.82.67:39469)
1025 20:08:04.198500 (+ 253us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.198504 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.198513 (+ 9us) server_negotiation.cc:408] Connection header received
1025 20:08:04.198573 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.198575 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.198621 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.198729 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:04.199086 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.199551 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.200131 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.200301 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.200350 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:04.200786 (+ 436us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:04.200886 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.200969 (+ 83us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.201010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20251025 20:08:04.209125 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.206856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:45611 (local address 127.18.82.65:43053)
1025 20:08:04.206963 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.206967 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.207081 (+ 114us) server_negotiation.cc:408] Connection header received
1025 20:08:04.207188 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.207191 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.207229 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.207283 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:04.207594 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.208041 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.208660 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.208773 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.208807 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:04.208864 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:04.208918 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.208992 (+ 74us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.209026 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 20:08:04.209143 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.206773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.18.82.67:45611)
1025 20:08:04.207017 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:04.207028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:04.207106 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:04.207297 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:04.207298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:04.207304 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:04.207520 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.207524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.208117 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.208120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:04.208571 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.208579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.208670 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.208702 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:04.208931 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:04.208934 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:08:04.208977 (+ 43us) client_negotiation.cc:240] Negotiation successful
1025 20:08:04.209026 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20251025 20:08:04.214422 20681 raft_consensus.cc:1275] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Refusing update from remote peer 2a3c9cf35e1541bbbb8a43a106f77448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:04.214526 20823 raft_consensus.cc:1275] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Refusing update from remote peer 2a3c9cf35e1541bbbb8a43a106f77448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:04.214928 21135 consensus_queue.cc:1048] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:04.215082 21130 consensus_queue.cc:1048] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:04.229732 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.227258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:56061 (local address 127.18.82.67:39469)
1025 20:08:04.227369 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:04.227373 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:04.227527 (+ 154us) server_negotiation.cc:408] Connection header received
1025 20:08:04.227620 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:04.227624 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:04.227675 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:04.227736 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:04.228132 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.228624 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.229243 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.229380 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.229415 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:04.229470 (+ 55us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:04.229557 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:04.229617 (+ 60us) server_negotiation.cc:300] Negotiation successful
1025 20:08:04.229651 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:08:04.229749 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:04.227204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:39469 (local address 127.18.82.65:56061)
1025 20:08:04.227444 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:04.227460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:04.227541 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:04.227779 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:04.227781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:04.227790 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:04.228030 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.228040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.228719 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:04.228722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:04.229163 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:04.229171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:04.229252 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:04.229272 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:04.229561 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:04.229565 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 20:08:04.229611 (+ 46us) client_negotiation.cc:240] Negotiation successful
1025 20:08:04.229661 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20251025 20:08:04.248262 20521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:04.248405 20521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:04.250308 20521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Sending DeleteTablet for 3 replicas of tablet 0d83cef3ac544c77b1b15a37513cd696
I20251025 20:08:04.250895 20803 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:40642
I20251025 20:08:04.250906 20945 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:49470
I20251025 20:08:04.251062 21114 tablet_replica.cc:333] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: stopping tablet replica
I20251025 20:08:04.251173 21114 raft_consensus.cc:2243] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:04.251200 21115 tablet_replica.cc:333] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: stopping tablet replica
I20251025 20:08:04.251294 21114 raft_consensus.cc:2272] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.251303 21115 raft_consensus.cc:2243] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:04.251559 21115 raft_consensus.cc:2272] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.251825 20661 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d83cef3ac544c77b1b15a37513cd696 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:04 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:51588
I20251025 20:08:04.251983 21113 tablet_replica.cc:333] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: stopping tablet replica
I20251025 20:08:04.252020 21114 ts_tablet_manager.cc:1916] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.252086 21113 raft_consensus.cc:2243] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:04.252116 21115 ts_tablet_manager.cc:1916] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.252167 21113 raft_consensus.cc:2272] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:04.252930 20521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:04.253298 20521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:04.254209 21114 ts_tablet_manager.cc:1929] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:04.254209 21115 ts_tablet_manager.cc:1929] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:04.254269 21115 log.cc:1199] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/0d83cef3ac544c77b1b15a37513cd696
I20251025 20:08:04.254269 21114 log.cc:1199] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/0d83cef3ac544c77b1b15a37513cd696
I20251025 20:08:04.254501 21113 ts_tablet_manager.cc:1916] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:04.254545 21115 ts_tablet_manager.cc:1950] T 0d83cef3ac544c77b1b15a37513cd696 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting consensus metadata
I20251025 20:08:04.254549 21114 ts_tablet_manager.cc:1950] T 0d83cef3ac544c77b1b15a37513cd696 P ca3676d39c3047de8df44c58f3c9898e: Deleting consensus metadata
I20251025 20:08:04.255115 20506 catalog_manager.cc:4985] TS 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469): tablet 0d83cef3ac544c77b1b15a37513cd696 (table test-table [id=576612433d6543a79ba6a6a3656e2017]) successfully deleted
I20251025 20:08:04.255271 20506 catalog_manager.cc:4985] TS ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995): tablet 0d83cef3ac544c77b1b15a37513cd696 (table test-table [id=576612433d6543a79ba6a6a3656e2017]) successfully deleted
I20251025 20:08:04.256783 21113 ts_tablet_manager.cc:1929] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:04.256848 21113 log.cc:1199] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/0d83cef3ac544c77b1b15a37513cd696
I20251025 20:08:04.257109 21113 ts_tablet_manager.cc:1950] T 0d83cef3ac544c77b1b15a37513cd696 P 219e0529732b42829f9789d9518a460d: Deleting consensus metadata
I20251025 20:08:04.257740 20507 catalog_manager.cc:4985] TS 219e0529732b42829f9789d9518a460d (127.18.82.65:43053): tablet 0d83cef3ac544c77b1b15a37513cd696 (table test-table [id=576612433d6543a79ba6a6a3656e2017]) successfully deleted
I20251025 20:08:04.259310 20661 tablet_service.cc:1505] Processing CreateTablet for tablet ee41f9d5c3b94a2186f64737281a7920 (DEFAULT_TABLE table=test-table [id=6d31ccf23dd24de3a3e515b61a60db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.259445 20803 tablet_service.cc:1505] Processing CreateTablet for tablet ee41f9d5c3b94a2186f64737281a7920 (DEFAULT_TABLE table=test-table [id=6d31ccf23dd24de3a3e515b61a60db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.259533 20945 tablet_service.cc:1505] Processing CreateTablet for tablet ee41f9d5c3b94a2186f64737281a7920 (DEFAULT_TABLE table=test-table [id=6d31ccf23dd24de3a3e515b61a60db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:04.259579 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee41f9d5c3b94a2186f64737281a7920. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.259676 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee41f9d5c3b94a2186f64737281a7920. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.259717 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee41f9d5c3b94a2186f64737281a7920. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:04.261421 21124 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: Bootstrap starting.
I20251025 20:08:04.261771 21126 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Bootstrap starting.
I20251025 20:08:04.262241 21124 tablet_bootstrap.cc:654] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.262586 21126 tablet_bootstrap.cc:654] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.263074 21125 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Bootstrap starting.
I20251025 20:08:04.263293 21124 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: No bootstrap required, opened a new log
I20251025 20:08:04.263406 21124 ts_tablet_manager.cc:1403] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:04.263743 21124 raft_consensus.cc:359] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.263832 21124 raft_consensus.cc:385] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.263880 21124 raft_consensus.cc:740] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219e0529732b42829f9789d9518a460d, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.263973 21125 tablet_bootstrap.cc:654] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:04.264029 21124 consensus_queue.cc:260] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.264279 21124 ts_tablet_manager.cc:1434] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 20:08:04.265062 21126 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: No bootstrap required, opened a new log
I20251025 20:08:04.265138 21126 ts_tablet_manager.cc:1403] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251025 20:08:04.265558 21126 raft_consensus.cc:359] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.265666 21126 raft_consensus.cc:385] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.265695 21126 raft_consensus.cc:740] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3676d39c3047de8df44c58f3c9898e, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.265794 21126 consensus_queue.cc:260] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.265946 21125 tablet_bootstrap.cc:492] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: No bootstrap required, opened a new log
I20251025 20:08:04.266010 21126 ts_tablet_manager.cc:1434] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:04.266014 21125 ts_tablet_manager.cc:1403] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:04.266515 21125 raft_consensus.cc:359] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.266634 21125 raft_consensus.cc:385] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:04.266680 21125 raft_consensus.cc:740] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Initialized, Role: FOLLOWER
I20251025 20:08:04.266809 21125 consensus_queue.cc:260] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.267066 21125 ts_tablet_manager.cc:1434] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:04.328029 21144 raft_consensus.cc:493] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:04.328150 21144 raft_consensus.cc:515] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.328585 21144 leader_election.cc:290] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995)
I20251025 20:08:04.328991 20822 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee41f9d5c3b94a2186f64737281a7920" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e" is_pre_election: true
I20251025 20:08:04.329006 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee41f9d5c3b94a2186f64737281a7920" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d" is_pre_election: true
I20251025 20:08:04.329154 20822 raft_consensus.cc:2468] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 0.
I20251025 20:08:04.329178 20681 raft_consensus.cc:2468] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 0.
I20251025 20:08:04.329454 20896 leader_election.cc:304] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c9cf35e1541bbbb8a43a106f77448, ca3676d39c3047de8df44c58f3c9898e; no voters:
I20251025 20:08:04.329653 21144 raft_consensus.cc:2804] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:04.329703 21144 raft_consensus.cc:493] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:04.329727 21144 raft_consensus.cc:3060] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.330574 21144 raft_consensus.cc:515] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.330890 21144 leader_election.cc:290] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 election: Requested vote from peers 219e0529732b42829f9789d9518a460d (127.18.82.65:43053), ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995)
I20251025 20:08:04.331171 20681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee41f9d5c3b94a2186f64737281a7920" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219e0529732b42829f9789d9518a460d"
I20251025 20:08:04.331214 20822 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee41f9d5c3b94a2186f64737281a7920" candidate_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3676d39c3047de8df44c58f3c9898e"
I20251025 20:08:04.331275 20681 raft_consensus.cc:3060] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.331321 20822 raft_consensus.cc:3060] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:04.332151 20681 raft_consensus.cc:2468] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 1.
I20251025 20:08:04.332335 20822 raft_consensus.cc:2468] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c9cf35e1541bbbb8a43a106f77448 in term 1.
I20251025 20:08:04.332422 20897 leader_election.cc:304] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219e0529732b42829f9789d9518a460d, 2a3c9cf35e1541bbbb8a43a106f77448; no voters:
I20251025 20:08:04.332625 21144 raft_consensus.cc:2804] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:04.332733 21144 raft_consensus.cc:697] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 LEADER]: Becoming Leader. State: Replica: 2a3c9cf35e1541bbbb8a43a106f77448, State: Running, Role: LEADER
I20251025 20:08:04.332878 21144 consensus_queue.cc:237] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } }
I20251025 20:08:04.334474 20521 catalog_manager.cc:5649] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67). New cstate: current_term: 1 leader_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a3c9cf35e1541bbbb8a43a106f77448" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 39469 } health_report { overall_health: HEALTHY } } }
I20251025 20:08:04.355599 20681 raft_consensus.cc:1275] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Refusing update from remote peer 2a3c9cf35e1541bbbb8a43a106f77448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:04.355741 20822 raft_consensus.cc:1275] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Refusing update from remote peer 2a3c9cf35e1541bbbb8a43a106f77448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:04.356040 21130 consensus_queue.cc:1048] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219e0529732b42829f9789d9518a460d" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:04.356151 21135 consensus_queue.cc:1048] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3676d39c3047de8df44c58f3c9898e" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:08:04.370294 21096 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251025 20:08:05.921922 20522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:05.922060 20522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:05.924217 20522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a68574ac58bb46b68f33c9bef4783bb6: Sending DeleteTablet for 3 replicas of tablet ee41f9d5c3b94a2186f64737281a7920
I20251025 20:08:05.924741 20661 tablet_service.cc:1552] Processing DeleteTablet for tablet ee41f9d5c3b94a2186f64737281a7920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:05 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:51588
I20251025 20:08:05.924747 20945 tablet_service.cc:1552] Processing DeleteTablet for tablet ee41f9d5c3b94a2186f64737281a7920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:05 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:49470
I20251025 20:08:05.924865 20803 tablet_service.cc:1552] Processing DeleteTablet for tablet ee41f9d5c3b94a2186f64737281a7920 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:05 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:40642
I20251025 20:08:05.925038 21176 tablet_replica.cc:333] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: stopping tablet replica
I20251025 20:08:05.925153 21177 tablet_replica.cc:333] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d: stopping tablet replica
I20251025 20:08:05.925159 21176 raft_consensus.cc:2243] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:05.925277 21177 raft_consensus.cc:2243] T ee41f9d5c3b94a2186f64737281a7920 P 219e0529732b42829f9789d9518a460d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:05.925336 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20596
I20251025 20:08:05.925339 21176 raft_consensus.cc:2272] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:05.927228 21176 ts_tablet_manager.cc:1916] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:05.929141 21176 ts_tablet_manager.cc:1929] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 20:08:05.929204 21176 log.cc:1199] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/ee41f9d5c3b94a2186f64737281a7920
I20251025 20:08:05.929486 21176 ts_tablet_manager.cc:1950] T ee41f9d5c3b94a2186f64737281a7920 P 2a3c9cf35e1541bbbb8a43a106f77448: Deleting consensus metadata
I20251025 20:08:05.929670 21178 tablet_replica.cc:333] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: stopping tablet replica
I20251025 20:08:05.929780 21178 raft_consensus.cc:2243] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:05.929888 21178 raft_consensus.cc:2272] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:05.930110 20506 catalog_manager.cc:4985] TS 2a3c9cf35e1541bbbb8a43a106f77448 (127.18.82.67:39469): tablet ee41f9d5c3b94a2186f64737281a7920 (table test-table [id=6d31ccf23dd24de3a3e515b61a60db06]) successfully deleted
I20251025 20:08:05.930408 21178 ts_tablet_manager.cc:1916] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:05.932090 21178 ts_tablet_manager.cc:1929] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 20:08:05.932147 21178 log.cc:1199] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/ee41f9d5c3b94a2186f64737281a7920
I20251025 20:08:05.932402 21178 ts_tablet_manager.cc:1950] T ee41f9d5c3b94a2186f64737281a7920 P ca3676d39c3047de8df44c58f3c9898e: Deleting consensus metadata
I20251025 20:08:05.932991 20506 catalog_manager.cc:4985] TS ca3676d39c3047de8df44c58f3c9898e (127.18.82.66:40995): tablet ee41f9d5c3b94a2186f64737281a7920 (table test-table [id=6d31ccf23dd24de3a3e515b61a60db06]) successfully deleted
W20251025 20:08:05.933766 20507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:43053 (error 108)
I20251025 20:08:05.934406 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20738
I20251025 20:08:05.935360 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:05.934358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43053 (local address 127.0.0.1:46058)
1025 20:08:05.935174 (+ 816us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:05.935244 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:43053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":66,"threads_started":1}
W20251025 20:08:05.935483 20507 catalog_manager.cc:4712] TS 219e0529732b42829f9789d9518a460d (127.18.82.65:43053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee41f9d5c3b94a2186f64737281a7920: Network error: Client connection negotiation failed: client connection to 127.18.82.65:43053: connect: Connection refused (error 111)
I20251025 20:08:05.941936 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20880
I20251025 20:08:05.948617 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 20490
2025-10-25T20:08:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6916 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:06 dist-test-slave-d2pk krb5kdc[21183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:06 dist-test-slave-d2pk krb5kdc[21183](info): set up 2 sockets
Oct 25 20:08:06 dist-test-slave-d2pk krb5kdc[21183](info): commencing operation
krb5kdc: starting...
W20251025 20:08:07.770308 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.799s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:07 dist-test-slave-d2pk krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:07.886648 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35799
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:35799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:07.983600 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:07.983785 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:07.983860 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:07.986655 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:07.986708 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:07.986747 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:07.986791 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:07.986833 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:07.990919 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:35799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:07.991850 21199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:07.992619 21199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:07.997838 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:07.997915 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:07.998050 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:07.997910 21199 server_base.cc:1047] running on GCE node
I20251025 20:08:07.998505 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:07.999332 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:08.000473 21199 hybrid_clock.cc:648] HybridClock initialized: now 1761422888000463 us; error 36 us; skew 500 ppm
Oct 25 20:08:07 dist-test-slave-d2pk krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422887, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:08.002872 21199 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:08.003810 21199 webserver.cc:492] Webserver started at http://127.18.82.126:43107/ using document root <none> and password file <none>
I20251025 20:08:08.004300 21199 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:08.004345 21199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:08.004515 21199 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:08.006083 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "bad37df90a7a4e66bb990a179e47b136"
format_stamp: "Formatted at 2025-10-25 20:08:07 on dist-test-slave-d2pk"
server_key: "2da0ec67377129bc66dfd4fbdddefc16"
server_key_iv: "34063ab15b31e0a08d0b19b35b786e17"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.006565 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "bad37df90a7a4e66bb990a179e47b136"
format_stamp: "Formatted at 2025-10-25 20:08:07 on dist-test-slave-d2pk"
server_key: "2da0ec67377129bc66dfd4fbdddefc16"
server_key_iv: "34063ab15b31e0a08d0b19b35b786e17"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.009749 21199 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:08.011651 21214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.012512 21199 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:08.012614 21199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "bad37df90a7a4e66bb990a179e47b136"
format_stamp: "Formatted at 2025-10-25 20:08:07 on dist-test-slave-d2pk"
server_key: "2da0ec67377129bc66dfd4fbdddefc16"
server_key_iv: "34063ab15b31e0a08d0b19b35b786e17"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.012698 21199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:08.039036 21199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:08.041383 21199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:08.041574 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:08.048314 21266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:35799 every 8 connection(s)
I20251025 20:08:08.048323 21199 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:35799
I20251025 20:08:08.049300 21199 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:08.051659 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21199
I20251025 20:08:08.051774 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:08.052008 18761 external_mini_cluster.cc:1468] Setting key 078ac64d1d5b03964cf5fed1f7f4d63c
I20251025 20:08:08.052066 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:08.058036 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136: Bootstrap starting.
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:08.060325 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:08.061057 21267 log.cc:826] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:08.062625 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136: No bootstrap required, opened a new log
I20251025 20:08:08.063601 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.053072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.18.82.126:35799)
1025 20:08:08.053389 (+ 317us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:08.053397 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:08.053425 (+ 28us) server_negotiation.cc:408] Connection header received
1025 20:08:08.054339 (+ 914us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:08.054355 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:08.054597 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:08.054829 (+ 232us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:08.055791 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.056487 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.057209 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.057364 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.059724 (+ 2360us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:08.059760 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:08.059769 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:08.059797 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:08.061411 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.061807 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.061811 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.061815 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.061880 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.062103 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.062106 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.062108 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.062388 (+ 280us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:08.062549 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:08.062834 (+ 285us) server_negotiation.cc:300] Negotiation successful
1025 20:08:08.063091 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20251025 20:08:08.065165 21267 raft_consensus.cc:359] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } }
I20251025 20:08:08.065385 21267 raft_consensus.cc:385] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:08.065485 21267 raft_consensus.cc:740] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad37df90a7a4e66bb990a179e47b136, State: Initialized, Role: FOLLOWER
I20251025 20:08:08.065882 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } }
I20251025 20:08:08.065989 21267 raft_consensus.cc:399] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:08.066030 21267 raft_consensus.cc:493] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:08.066087 21267 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:08.066840 21267 raft_consensus.cc:515] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } }
I20251025 20:08:08.067109 21267 leader_election.cc:304] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bad37df90a7a4e66bb990a179e47b136; no voters:
I20251025 20:08:08.067327 21267 leader_election.cc:290] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:08.067677 21272 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:08.067845 21272 raft_consensus.cc:697] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [term 1 LEADER]: Becoming Leader. State: Replica: bad37df90a7a4e66bb990a179e47b136, State: Running, Role: LEADER
I20251025 20:08:08.068087 21272 consensus_queue.cc:237] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } }
I20251025 20:08:08.068331 21267 sys_catalog.cc:565] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:08.069403 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bad37df90a7a4e66bb990a179e47b136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } } }
I20251025 20:08:08.069545 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:08.069891 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bad37df90a7a4e66bb990a179e47b136. Latest consensus state: current_term: 1 leader_uuid: "bad37df90a7a4e66bb990a179e47b136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad37df90a7a4e66bb990a179e47b136" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35799 } } }
I20251025 20:08:08.069923 21281 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:08.069977 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:08.072649 21281 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:08.077095 21281 catalog_manager.cc:1357] Generated new cluster ID: 3405735718524464963d559da4eba5fe
I20251025 20:08:08.077153 21281 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:08.092209 21281 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:08.093029 21281 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:08.103866 21281 catalog_manager.cc:6022] T 00000000000000000000000000000000 P bad37df90a7a4e66bb990a179e47b136: Generated new TSK 0
I20251025 20:08:08.104468 21281 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:08.148828 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35799
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:08.245230 21295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:08.245424 21295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:08.245528 21295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:08.248929 21295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:08.248986 21295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:08.249086 21295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:08.253016 21295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:08.254029 21295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:08.254791 21295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:08.260489 21301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.260486 21300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.260538 21303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:08.260830 21295 server_base.cc:1047] running on GCE node
I20251025 20:08:08.261127 21295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:08.261624 21295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:08.262763 21295 hybrid_clock.cc:648] HybridClock initialized: now 1761422888262752 us; error 28 us; skew 500 ppm
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:08.264911 21295 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:08.265862 21295 webserver.cc:492] Webserver started at http://127.18.82.65:45935/ using document root <none> and password file <none>
I20251025 20:08:08.266409 21295 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:08.266469 21295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:08.266641 21295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:08.268172 21295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "f1fac16dd0bc411183cb3bb543ee0b89"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a3f7eb968f46f641f2ee8fa2391dccee"
server_key_iv: "9157e87b4e150534b43791894a8f4bc2"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.268671 21295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "f1fac16dd0bc411183cb3bb543ee0b89"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a3f7eb968f46f641f2ee8fa2391dccee"
server_key_iv: "9157e87b4e150534b43791894a8f4bc2"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.271771 21295 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 20:08:08.273728 21310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.274622 21295 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:08.274717 21295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "f1fac16dd0bc411183cb3bb543ee0b89"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a3f7eb968f46f641f2ee8fa2391dccee"
server_key_iv: "9157e87b4e150534b43791894a8f4bc2"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.274794 21295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:08.291177 21295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:08.293591 21295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:08.293746 21295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:08.294216 21295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:08.294953 21295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:08.294998 21295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.295053 21295 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:08.295099 21295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.303220 21295 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:37497
I20251025 20:08:08.303274 21423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:37497 every 8 connection(s)
I20251025 20:08:08.304056 21295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:08.313570 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21295
I20251025 20:08:08.313674 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:08.313936 18761 external_mini_cluster.cc:1468] Setting key 89ddc1bca56cdc6bd8c4a5881337e6c4
I20251025 20:08:08.314865 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.305629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:43085 (local address 127.18.82.126:35799)
1025 20:08:08.305756 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:08.305760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:08.306323 (+ 563us) server_negotiation.cc:408] Connection header received
1025 20:08:08.307137 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:08.307141 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:08.307203 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:08.307277 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:08.308793 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.309285 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.309937 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.310101 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.311852 (+ 1751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:08.311871 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:08.311873 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:08.311893 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:08.313022 (+ 1129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.313474 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.313478 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.313479 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.313569 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.313868 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.313871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.313872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.314024 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:08.314098 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:08.314596 (+ 498us) server_negotiation.cc:300] Negotiation successful
1025 20:08:08.314714 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:08:08.315416 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.305813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35799 (local address 127.18.82.65:43085)
1025 20:08:08.306190 (+ 377us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:08.306220 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:08.306945 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:08.307439 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:08.307446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:08.307786 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:08.308629 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.308651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.309406 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.309411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:08.309835 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.309844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.310023 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.310527 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:08.310549 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:08.311729 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:08.313129 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.313134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.313144 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.313378 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.313672 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.313675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.313677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.313781 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.314109 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:08.314115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:08.314347 (+ 232us) client_negotiation.cc:769] Sending connection context
1025 20:08:08.314532 (+ 185us) client_negotiation.cc:240] Negotiation successful
1025 20:08:08.314743 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20251025 20:08:08.316418 21424 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35799
I20251025 20:08:08.316623 21424 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:08.317170 21424 heartbeater.cc:507] Master 127.18.82.126:35799 requested a full tablet report, sending...
I20251025 20:08:08.318660 21231 ts_manager.cc:194] Registered new tserver with Master: f1fac16dd0bc411183cb3bb543ee0b89 (127.18.82.65:37497)
I20251025 20:08:08.319784 21231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:43085
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:08.355983 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35799
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:08.452708 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:08.452873 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:08.452919 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:08.455649 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:08.455696 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:08.455760 21431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:08.459280 21431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:08.460259 21431 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:08.460984 21431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:08.467034 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.467041 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.467063 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:08.467231 21431 server_base.cc:1047] running on GCE node
I20251025 20:08:08.467686 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:08.468148 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:08.469285 21431 hybrid_clock.cc:648] HybridClock initialized: now 1761422888469253 us; error 49 us; skew 500 ppm
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:08.471707 21431 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:08.472579 21431 webserver.cc:492] Webserver started at http://127.18.82.66:37337/ using document root <none> and password file <none>
I20251025 20:08:08.473052 21431 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:08.473100 21431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:08.473281 21431 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:08.474851 21431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "bd20e90dcb5f4986a9ea579cfa576590"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a0c6c09e955fbe9814dac403a52666e9"
server_key_iv: "594047a1c7c7d655eb22df3f8e1a5477"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.475267 21431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "bd20e90dcb5f4986a9ea579cfa576590"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a0c6c09e955fbe9814dac403a52666e9"
server_key_iv: "594047a1c7c7d655eb22df3f8e1a5477"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.478245 21431 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:08.480319 21446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.481212 21431 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:08.481307 21431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "bd20e90dcb5f4986a9ea579cfa576590"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "a0c6c09e955fbe9814dac403a52666e9"
server_key_iv: "594047a1c7c7d655eb22df3f8e1a5477"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.481381 21431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:08.501600 21431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:08.504020 21431 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:08.504181 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:08.504666 21431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:08.505436 21431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:08.505482 21431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.505553 21431 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:08.505584 21431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.514209 21431 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:41255
I20251025 20:08:08.514245 21559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:41255 every 8 connection(s)
I20251025 20:08:08.515074 21431 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:08.520319 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21431
I20251025 20:08:08.520439 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:08.520661 18761 external_mini_cluster.cc:1468] Setting key 8aeceab4bf7594b23ef0ee298f0c4cc3
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:08.526542 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.516623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:38091 (local address 127.18.82.126:35799)
1025 20:08:08.516760 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:08.516763 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:08.517414 (+ 651us) server_negotiation.cc:408] Connection header received
1025 20:08:08.518241 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:08.518244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:08.518329 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:08.518419 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:08.519975 (+ 1556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.520606 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.521275 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.521437 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.523432 (+ 1995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:08.523447 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:08.523450 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:08.523472 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:08.524707 (+ 1235us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.525167 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.525170 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.525171 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.525211 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.525454 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.525457 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.525459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.525636 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:08.525720 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:08.526232 (+ 512us) server_negotiation.cc:300] Negotiation successful
1025 20:08:08.526386 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 20:08:08.527096 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.516908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35799 (local address 127.18.82.66:38091)
1025 20:08:08.517291 (+ 383us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:08.517323 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:08.518062 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:08.518586 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:08.518594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:08.518944 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:08.519829 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.519843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.520741 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.520744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:08.521184 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.521192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.521392 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.522012 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:08.522042 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:08.523296 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:08.524814 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.524821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.524833 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.525073 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.525288 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.525290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.525292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.525366 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.525732 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:08.525740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:08.525972 (+ 232us) client_negotiation.cc:769] Sending connection context
1025 20:08:08.526185 (+ 213us) client_negotiation.cc:240] Negotiation successful
1025 20:08:08.526477 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20251025 20:08:08.528112 21560 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35799
I20251025 20:08:08.528355 21560 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:08:08.528857 21560 heartbeater.cc:507] Master 127.18.82.126:35799 requested a full tablet report, sending...
I20251025 20:08:08.529889 21231 ts_manager.cc:194] Registered new tserver with Master: bd20e90dcb5f4986a9ea579cfa576590 (127.18.82.66:41255)
I20251025 20:08:08.530490 21231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:38091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:08.559876 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35799
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:08.657737 21567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:08.657898 21567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:08.657946 21567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:08.660822 21567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:08.660866 21567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:08.660933 21567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:08.664609 21567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:08.665526 21567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:08.666350 21567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:08.672403 21573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.672457 21575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:08.672403 21572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:08.672552 21567 server_base.cc:1047] running on GCE node
I20251025 20:08:08.672912 21567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:08.673343 21567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:08.674479 21567 hybrid_clock.cc:648] HybridClock initialized: now 1761422888674466 us; error 24 us; skew 500 ppm
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:08.676913 21567 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:08.677848 21567 webserver.cc:492] Webserver started at http://127.18.82.67:46605/ using document root <none> and password file <none>
I20251025 20:08:08.678396 21567 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:08.678457 21567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:08.678629 21567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:08.680197 21567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "1437028de9df49a5a96a560756d4daed"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "eec681be7ba3cd2ca1fbf793a2870857"
server_key_iv: "38852dce5d7a0dd60a58f26eee166e29"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.680637 21567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "1437028de9df49a5a96a560756d4daed"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "eec681be7ba3cd2ca1fbf793a2870857"
server_key_iv: "38852dce5d7a0dd60a58f26eee166e29"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.683688 21567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251025 20:08:08.685611 21582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.686488 21567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:08.686589 21567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "1437028de9df49a5a96a560756d4daed"
format_stamp: "Formatted at 2025-10-25 20:08:08 on dist-test-slave-d2pk"
server_key: "eec681be7ba3cd2ca1fbf793a2870857"
server_key_iv: "38852dce5d7a0dd60a58f26eee166e29"
server_key_version: "encryptionkey@0"
I20251025 20:08:08.686695 21567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:08.701860 21567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:08.704032 21567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:08.704190 21567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:08.704679 21567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:08.705435 21567 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:08.705477 21567 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.705547 21567 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:08.705572 21567 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:08.713775 21567 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:45617
I20251025 20:08:08.713809 21695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:45617 every 8 connection(s)
I20251025 20:08:08.714582 21567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:08.724669 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21567
I20251025 20:08:08.724757 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:08.724958 18761 external_mini_cluster.cc:1468] Setting key c4ecab945189e7068bd1ddb988ad227d
I20251025 20:08:08.724982 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.716098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:33635 (local address 127.18.82.126:35799)
1025 20:08:08.716231 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:08.716236 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:08.716840 (+ 604us) server_negotiation.cc:408] Connection header received
1025 20:08:08.717645 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:08.717648 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:08.717695 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:08.717760 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:08.719252 (+ 1492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.719732 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.720209 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.720310 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.722226 (+ 1916us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:08.722243 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:08.722245 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:08.722267 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:08.723330 (+ 1063us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.723778 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.723781 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.723782 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.723824 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:08.724079 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:08.724094 (+ 15us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:08.724095 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:08.724232 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:08.724313 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:08.724728 (+ 415us) server_negotiation.cc:300] Negotiation successful
1025 20:08:08.724837 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:08:08.725577 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:08.716335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35799 (local address 127.18.82.67:33635)
1025 20:08:08.716717 (+ 382us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:08.716749 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:08.717439 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:08.717901 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:08.717908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:08.718224 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:08.719124 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.719137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.719807 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:08.719809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:08.720140 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:08.720145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:08.720284 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:08.720770 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:08.720790 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:08.722091 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:08.723438 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.723443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.723452 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.723685 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.723899 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:08.723903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:08.723905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:08.723994 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:08.724326 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:08.724333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:08.724518 (+ 185us) client_negotiation.cc:769] Sending connection context
1025 20:08:08.724676 (+ 158us) client_negotiation.cc:240] Negotiation successful
1025 20:08:08.724891 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20251025 20:08:08.726529 21696 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35799
I20251025 20:08:08.726755 21696 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:08.727188 21696 heartbeater.cc:507] Master 127.18.82.126:35799 requested a full tablet report, sending...
I20251025 20:08:08.728048 21231 ts_manager.cc:194] Registered new tserver with Master: 1437028de9df49a5a96a560756d4daed (127.18.82.67:45617)
I20251025 20:08:08.728556 21231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:33635
I20251025 20:08:08.738646 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 20:08:08.749123 21270 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:08.747882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.18.82.126:35799)
1025 20:08:08.747996 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:08.748001 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:08.748140 (+ 139us) server_negotiation.cc:408] Connection header received
1025 20:08:08.748252 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:08.748255 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:08.748301 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:08.748377 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:08.749009 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47252: BlockingRecv error: recv got EOF from 127.0.0.1:47252 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 20:08:08.750283 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21295
I20251025 20:08:08.756322 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21431
I20251025 20:08:08.761801 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21567
I20251025 20:08:08.766836 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21199
2025-10-25T20:08:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2817 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21708](info): set up 2 sockets
Oct 25 20:08:08 dist-test-slave-d2pk krb5kdc[21708](info): commencing operation
krb5kdc: starting...
W20251025 20:08:10.610107 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/test-user.keytab.
W20251025 20:08:10.865217 18761 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.146s user 0.001s sys 0.001s
Oct 25 20:08:10 dist-test-slave-d2pk krb5kdc[21708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
W20251025 20:08:11.476765 18761 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.18.82.126: real 0.593s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
W20251025 20:08:12.232311 18761 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.18.82.126: real 0.755s user 0.000s sys 0.003s
I20251025 20:08:12.233014 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:39747
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:12.338521 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:12.338765 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:12.338843 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:12.341929 21724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:12.341979 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:12.342015 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:12.342056 21724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:12.342087 21724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:12.346004 21724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:12.347065 21724 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:12.348019 21724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:12.353382 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:12.353392 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:12.353389 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:12.353561 21724 server_base.cc:1047] running on GCE node
I20251025 20:08:12.354043 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:12.354871 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:12.356016 21724 hybrid_clock.cc:648] HybridClock initialized: now 1761422892356006 us; error 23 us; skew 500 ppm
Oct 25 20:08:12 dist-test-slave-d2pk krb5kdc[21708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422892, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:12.358685 21724 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:12.359701 21724 webserver.cc:492] Webserver started at http://127.18.82.126:46729/ using document root <none> and password file <none>
I20251025 20:08:12.360214 21724 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:12.360262 21724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:12.360424 21724 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:12.362041 21724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "0d131dfa2c2148568e7c653395ec9807"
format_stamp: "Formatted at 2025-10-25 20:08:12 on dist-test-slave-d2pk"
server_key: "6c3ea401d561268a4bbf316a87ba98ed"
server_key_iv: "1df1cfc006307e2612c8faa76264fabe"
server_key_version: "encryptionkey@0"
I20251025 20:08:12.362536 21724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "0d131dfa2c2148568e7c653395ec9807"
format_stamp: "Formatted at 2025-10-25 20:08:12 on dist-test-slave-d2pk"
server_key: "6c3ea401d561268a4bbf316a87ba98ed"
server_key_iv: "1df1cfc006307e2612c8faa76264fabe"
server_key_version: "encryptionkey@0"
I20251025 20:08:12.365609 21724 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:12.367576 21739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:12.368595 21724 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:12.368726 21724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "0d131dfa2c2148568e7c653395ec9807"
format_stamp: "Formatted at 2025-10-25 20:08:12 on dist-test-slave-d2pk"
server_key: "6c3ea401d561268a4bbf316a87ba98ed"
server_key_iv: "1df1cfc006307e2612c8faa76264fabe"
server_key_version: "encryptionkey@0"
I20251025 20:08:12.368808 21724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:12.384661 21724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:12.390594 21724 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:12.390774 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:12.397472 21724 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:39747
I20251025 20:08:12.397495 21791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:39747 every 8 connection(s)
I20251025 20:08:12.398454 21724 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:12.401139 21792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:12.406595 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: Bootstrap starting.
I20251025 20:08:12.408655 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21724
I20251025 20:08:12.408804 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:12.409000 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:12.409010 18761 external_mini_cluster.cc:1468] Setting key 46148e2bff4b0ca061951b40ad90b2c7
I20251025 20:08:12.410153 21792 log.cc:826] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:12.412169 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: No bootstrap required, opened a new log
I20251025 20:08:12.415190 21792 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } }
I20251025 20:08:12.415366 21792 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:12.415398 21792 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d131dfa2c2148568e7c653395ec9807, State: Initialized, Role: FOLLOWER
I20251025 20:08:12.415807 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } }
I20251025 20:08:12.415942 21792 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:12.416013 21792 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:12.416093 21792 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 0 FOLLOWER]: Advancing to term 1
Oct 25 20:08:12 dist-test-slave-d2pk krb5kdc[21708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:12.420908 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:12.410066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.18.82.126:39747)
1025 20:08:12.410500 (+ 434us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:12.410512 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:12.410553 (+ 41us) server_negotiation.cc:408] Connection header received
1025 20:08:12.411186 (+ 633us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:12.411209 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:12.411554 (+ 345us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:12.411882 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:12.412719 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:12.413913 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:12.414518 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:12.414801 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:12.416925 (+ 2124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:12.416943 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:12.416952 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:12.416980 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:12.418613 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:12.418990 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:12.418994 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:12.418998 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:12.419061 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:12.419213 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:12.419215 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:12.419216 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:12.419513 (+ 297us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:12.419650 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:12.419840 (+ 190us) server_negotiation.cc:300] Negotiation successful
1025 20:08:12.420082 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20251025 20:08:12.424382 21792 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } }
I20251025 20:08:12.424728 21792 leader_election.cc:304] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d131dfa2c2148568e7c653395ec9807; no voters:
I20251025 20:08:12.425016 21792 leader_election.cc:290] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:12.425112 21797 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:12.425314 21797 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [term 1 LEADER]: Becoming Leader. State: Replica: 0d131dfa2c2148568e7c653395ec9807, State: Running, Role: LEADER
I20251025 20:08:12.425673 21797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } }
I20251025 20:08:12.426263 21792 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:12.427076 21799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d131dfa2c2148568e7c653395ec9807. Latest consensus state: current_term: 1 leader_uuid: "0d131dfa2c2148568e7c653395ec9807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } } }
I20251025 20:08:12.427191 21799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:12.427466 21798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d131dfa2c2148568e7c653395ec9807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d131dfa2c2148568e7c653395ec9807" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39747 } } }
I20251025 20:08:12.427568 21798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807 [sys.catalog]: This master's current role is: LEADER
W20251025 20:08:12.431272 21812 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:12.431335 21812 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:12.431406 21806 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:12.432044 21806 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:12.437083 21806 catalog_manager.cc:1357] Generated new cluster ID: e9dd5661205441538a7cd8272683aae7
I20251025 20:08:12.437141 21806 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:12.454316 21806 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:12.455377 21806 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:12.461802 21806 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0d131dfa2c2148568e7c653395ec9807: Generated new TSK 0
I20251025 20:08:12.462414 21806 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
W20251025 20:08:13.367556 18761 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.18.82.65: real 0.893s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
W20251025 20:08:14.341143 18761 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.18.82.65: real 0.973s user 0.003s sys 0.000s
I20251025 20:08:14.341709 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:39747
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:14.448602 21820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:14.448849 21820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:14.448938 21820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:14.452162 21820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:14.452227 21820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:14.452327 21820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:14.456497 21820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:14.457489 21820 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:14.458299 21820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:14.464740 21828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:14.464754 21825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:14.464921 21820 server_base.cc:1047] running on GCE node
W20251025 20:08:14.464754 21826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:14.465234 21820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:14.465799 21820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:14.466943 21820 hybrid_clock.cc:648] HybridClock initialized: now 1761422894466931 us; error 24 us; skew 500 ppm
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:14.469496 21820 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:14.470485 21820 webserver.cc:492] Webserver started at http://127.18.82.65:39381/ using document root <none> and password file <none>
I20251025 20:08:14.470974 21820 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:14.471017 21820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:14.471163 21820 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:14.472824 21820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "ffabc9f3268546458da7aec1dc25a44f"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "02aa3eaf851908de61f75fa6d5c320d5"
server_key_iv: "478b5c76e876c753e98b50bc817a0212"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.473279 21820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "ffabc9f3268546458da7aec1dc25a44f"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "02aa3eaf851908de61f75fa6d5c320d5"
server_key_iv: "478b5c76e876c753e98b50bc817a0212"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.476608 21820 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:14.478685 21835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.479647 21820 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:14.479736 21820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "ffabc9f3268546458da7aec1dc25a44f"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "02aa3eaf851908de61f75fa6d5c320d5"
server_key_iv: "478b5c76e876c753e98b50bc817a0212"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.479807 21820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:14.493321 21820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:14.495918 21820 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:14.496073 21820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:14.496527 21820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:14.497329 21820 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:14.497375 21820 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.497426 21820 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:14.497447 21820 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.505792 21820 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:38123
I20251025 20:08:14.505823 21948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:38123 every 8 connection(s)
I20251025 20:08:14.506747 21820 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:14.506979 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21820
I20251025 20:08:14.507102 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:14.507400 18761 external_mini_cluster.cc:1468] Setting key 28801485af3322f44bdd758cffe90aff
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
I20251025 20:08:14.519381 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.508725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:33597 (local address 127.18.82.126:39747)
1025 20:08:14.508998 (+ 273us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:14.509003 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:14.509684 (+ 681us) server_negotiation.cc:408] Connection header received
1025 20:08:14.510505 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:14.510509 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:14.510566 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:14.510655 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:14.511862 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.512625 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.513368 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.513608 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.516012 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:14.516039 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:14.516045 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:14.516074 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:14.517795 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.518228 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.518231 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.518232 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.518287 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.518482 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.518485 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.518486 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.518626 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:14.518716 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:14.519162 (+ 446us) server_negotiation.cc:300] Negotiation successful
1025 20:08:14.519252 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20251025 20:08:14.519982 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.509074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:39747 (local address 127.18.82.65:33597)
1025 20:08:14.509559 (+ 485us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:14.509596 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:14.510305 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:14.510826 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:14.510834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:14.511191 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:14.511711 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.511723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.512769 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.512773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:14.513226 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.513236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.513823 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.514459 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:14.514480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:14.515858 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:14.517915 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.517919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.517928 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.518142 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.518348 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.518350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.518351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.518417 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.518715 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:14.518720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:14.518965 (+ 245us) client_negotiation.cc:769] Sending connection context
1025 20:08:14.519135 (+ 170us) client_negotiation.cc:240] Negotiation successful
1025 20:08:14.519336 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20251025 20:08:14.521046 21949 heartbeater.cc:344] Connected to a master server at 127.18.82.126:39747
I20251025 20:08:14.521278 21949 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:14.521800 21949 heartbeater.cc:507] Master 127.18.82.126:39747 requested a full tablet report, sending...
I20251025 20:08:14.523241 21756 ts_manager.cc:194] Registered new tserver with Master: ffabc9f3268546458da7aec1dc25a44f (127.18.82.65:38123)
I20251025 20:08:14.524323 21756 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:33597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:14.576032 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:39747
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:14.676090 21957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:14.676277 21957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:14.676355 21957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:14.679243 21957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:14.679299 21957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:14.679407 21957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:14.683072 21957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:14.683995 21957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:14.684775 21957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:14.690765 21962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:14.690765 21965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:14.690850 21963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:14.691167 21957 server_base.cc:1047] running on GCE node
I20251025 20:08:14.691527 21957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:14.692097 21957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:14.693254 21957 hybrid_clock.cc:648] HybridClock initialized: now 1761422894693236 us; error 36 us; skew 500 ppm
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:14.696519 21957 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:14.697460 21957 webserver.cc:492] Webserver started at http://127.18.82.66:44925/ using document root <none> and password file <none>
I20251025 20:08:14.698001 21957 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:14.698051 21957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:14.698230 21957 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:14.699841 21957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "3361c64279c44b80b3f530ab20c36edd"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "b868eed03214881c4d06f9a16795663c"
server_key_iv: "c46ac692ac696a9c60b2f6297885bce9"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.700297 21957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "3361c64279c44b80b3f530ab20c36edd"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "b868eed03214881c4d06f9a16795663c"
server_key_iv: "c46ac692ac696a9c60b2f6297885bce9"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.703467 21957 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:14.705526 21972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.706434 21957 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:14.706535 21957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "3361c64279c44b80b3f530ab20c36edd"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "b868eed03214881c4d06f9a16795663c"
server_key_iv: "c46ac692ac696a9c60b2f6297885bce9"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.706615 21957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:14.725844 21957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:14.728665 21957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:14.728842 21957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:14.729352 21957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:14.730216 21957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:14.730257 21957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.730310 21957 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:14.730341 21957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.737821 21957 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:43509
I20251025 20:08:14.737854 22085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:43509 every 8 connection(s)
I20251025 20:08:14.738620 21957 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:14.740888 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 21957
I20251025 20:08:14.741009 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:14.741298 18761 external_mini_cluster.cc:1468] Setting key 9242c4fa183ea236672cd38b4dbf4c16
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:08:14.750526 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.740224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:60815 (local address 127.18.82.126:39747)
1025 20:08:14.740386 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:14.740391 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:14.741131 (+ 740us) server_negotiation.cc:408] Connection header received
1025 20:08:14.742111 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:14.742114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:14.742167 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:14.742235 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:14.743690 (+ 1455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.744137 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.744732 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.744936 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.747318 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:14.747336 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:14.747338 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:14.747361 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:14.748737 (+ 1376us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.749177 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.749181 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.749182 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.749226 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.749533 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.749537 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.749539 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.749690 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:14.749785 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:14.750258 (+ 473us) server_negotiation.cc:300] Negotiation successful
1025 20:08:14.750372 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:14.751034 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.740474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:39747 (local address 127.18.82.66:60815)
1025 20:08:14.740940 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:14.741024 (+ 84us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:14.741925 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:14.742407 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:14.742416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:14.742867 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:14.743543 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.743562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.744264 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.744268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:14.744623 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.744630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.745237 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.745783 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:14.745805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:14.747179 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:14.748838 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.748843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.748853 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.749080 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.749307 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.749312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.749314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.749422 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.749803 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:14.749809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:14.750019 (+ 210us) client_negotiation.cc:769] Sending connection context
1025 20:08:14.750197 (+ 178us) client_negotiation.cc:240] Negotiation successful
1025 20:08:14.750450 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
I20251025 20:08:14.751993 22086 heartbeater.cc:344] Connected to a master server at 127.18.82.126:39747
I20251025 20:08:14.752214 22086 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:14.752663 22086 heartbeater.cc:507] Master 127.18.82.126:39747 requested a full tablet report, sending...
I20251025 20:08:14.753638 21756 ts_manager.cc:194] Registered new tserver with Master: 3361c64279c44b80b3f530ab20c36edd (127.18.82.66:43509)
I20251025 20:08:14.754383 21756 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:60815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:14.782327 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:39747
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:14.877807 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:14.878000 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:14.878047 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:14.880995 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:14.881038 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:14.881104 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:14.884585 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:14.885409 22093 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:14.886147 22093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:14.891953 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:14.891960 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:14.891986 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:14.892324 22093 server_base.cc:1047] running on GCE node
I20251025 20:08:14.892625 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:14.893074 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:14.894202 22093 hybrid_clock.cc:648] HybridClock initialized: now 1761422894894182 us; error 33 us; skew 500 ppm
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:14.896829 22093 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:14.897775 22093 webserver.cc:492] Webserver started at http://127.18.82.67:43863/ using document root <none> and password file <none>
I20251025 20:08:14.898221 22093 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:14.898264 22093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:14.898411 22093 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:14.899942 22093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "303f6f22258e438faac04907e2d741a7"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "33c7b110329caeaaf8f58f8f0ae4c083"
server_key_iv: "6172e2f78109771f10b7c4f27a32253c"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.900364 22093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "303f6f22258e438faac04907e2d741a7"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "33c7b110329caeaaf8f58f8f0ae4c083"
server_key_iv: "6172e2f78109771f10b7c4f27a32253c"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.903362 22093 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:14.905272 22108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.906185 22093 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:14.906277 22093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "303f6f22258e438faac04907e2d741a7"
format_stamp: "Formatted at 2025-10-25 20:08:14 on dist-test-slave-d2pk"
server_key: "33c7b110329caeaaf8f58f8f0ae4c083"
server_key_iv: "6172e2f78109771f10b7c4f27a32253c"
server_key_version: "encryptionkey@0"
I20251025 20:08:14.906399 22093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:14.928766 22093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:14.931581 22093 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:14.931715 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:14.932193 22093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:14.932974 22093 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:14.933017 22093 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.933058 22093 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:14.933089 22093 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:14.941102 22093 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34875
I20251025 20:08:14.941136 22221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34875 every 8 connection(s)
I20251025 20:08:14.941977 22093 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:14.947579 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22093
I20251025 20:08:14.947721 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:14.947990 18761 external_mini_cluster.cc:1468] Setting key 19ed9b3a18b68480d2dfa5a520ceeaa9
Oct 25 20:08:14 dist-test-slave-d2pk krb5kdc[21708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422894, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:14.952816 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.943545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:43855 (local address 127.18.82.126:39747)
1025 20:08:14.943683 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:14.943687 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:14.944266 (+ 579us) server_negotiation.cc:408] Connection header received
1025 20:08:14.944957 (+ 691us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:14.944960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:14.945007 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:14.945074 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:14.946236 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.946770 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.947346 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.947465 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.950198 (+ 2733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:14.950225 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:14.950227 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:14.950249 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:14.951489 (+ 1240us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.951874 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.951877 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.951878 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.951910 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:14.952082 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:14.952084 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:14.952085 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:14.952202 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:14.952263 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:14.952619 (+ 356us) server_negotiation.cc:300] Negotiation successful
1025 20:08:14.952710 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:08:14.953375 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:14.943744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:39747 (local address 127.18.82.67:43855)
1025 20:08:14.944128 (+ 384us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:14.944159 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:14.944795 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:14.945197 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:14.945204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:14.945605 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:14.946116 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.946127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.946859 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:14.946861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:14.947234 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:14.947239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:14.947983 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:14.948497 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:14.948517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:14.950062 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:14.951601 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.951604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.951611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.951807 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.951966 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:14.951968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:14.951969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:14.952028 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:14.952262 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:14.952266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:14.952451 (+ 185us) client_negotiation.cc:769] Sending connection context
1025 20:08:14.952594 (+ 143us) client_negotiation.cc:240] Negotiation successful
1025 20:08:14.952774 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20251025 20:08:14.954350 22222 heartbeater.cc:344] Connected to a master server at 127.18.82.126:39747
I20251025 20:08:14.954584 22222 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:14.955006 22222 heartbeater.cc:507] Master 127.18.82.126:39747 requested a full tablet report, sending...
I20251025 20:08:14.955885 21756 ts_manager.cc:194] Registered new tserver with Master: 303f6f22258e438faac04907e2d741a7 (127.18.82.67:34875)
I20251025 20:08:14.956446 21756 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:43855
I20251025 20:08:14.961550 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 20:08:15.080075 21951 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:15.077033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.18.82.126:39747)
1025 20:08:15.077164 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:15.077168 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:15.078009 (+ 841us) server_negotiation.cc:408] Connection header received
1025 20:08:15.078729 (+ 720us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:15.078732 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:15.078779 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:15.078849 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:15.079946 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37884: BlockingRecv error: recv got EOF from 127.0.0.1:37884 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20251025 20:08:15.081365 21951 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:15.080711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.18.82.126:39747)
1025 20:08:15.080825 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:15.080829 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:15.080841 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:15.080919 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:15.080922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:15.080957 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:15.081010 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:15.081276 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37886: BlockingRecv error: recv got EOF from 127.0.0.1:37886 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20251025 20:08:15.082307 21951 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:15.081628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.18.82.126:39747)
1025 20:08:15.081726 (+ 98us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:15.081729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:15.081839 (+ 110us) server_negotiation.cc:408] Connection header received
1025 20:08:15.081901 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:15.081903 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:15.081934 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:15.081991 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:15.082231 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37898: BlockingRecv error: recv got EOF from 127.0.0.1:37898 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20251025 20:08:15.085692 21951 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:15.084871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.18.82.126:39747)
1025 20:08:15.084981 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:15.084986 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:15.085045 (+ 59us) server_negotiation.cc:408] Connection header received
1025 20:08:15.085151 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:15.085154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:15.085187 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:15.085245 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:15.085619 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37906: BlockingRecv error: recv got EOF from 127.0.0.1:37906 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 20:08:15.094221 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21820
I20251025 20:08:15.099961 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21957
I20251025 20:08:15.104758 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22093
I20251025 20:08:15.109750 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 21724
2025-10-25T20:08:15Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (6342 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:15 dist-test-slave-d2pk krb5kdc[22245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:15 dist-test-slave-d2pk krb5kdc[22245](info): set up 2 sockets
Oct 25 20:08:15 dist-test-slave-d2pk krb5kdc[22245](info): commencing operation
krb5kdc: starting...
W20251025 20:08:16.946205 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:17.070421 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43933
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:43933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:17.169077 22261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:17.169277 22261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:17.169327 22261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:17.172243 22261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:17.172293 22261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:17.172312 22261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:17.172346 22261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:17.172366 22261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:17.175889 22261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:43933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:17.176756 22261 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:17.177474 22261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:17.182512 22269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:17.182520 22267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:17.182797 22266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:17.182801 22261 server_base.cc:1047] running on GCE node
I20251025 20:08:17.183249 22261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:17.184038 22261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:17.185185 22261 hybrid_clock.cc:648] HybridClock initialized: now 1761422897185165 us; error 40 us; skew 500 ppm
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422897, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:17.187582 22261 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:17.188457 22261 webserver.cc:492] Webserver started at http://127.18.82.126:41547/ using document root <none> and password file <none>
I20251025 20:08:17.188915 22261 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:17.188954 22261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:17.189091 22261 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:17.190745 22261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "b01df4be952242ee9cf62067c18fa3a9"
format_stamp: "Formatted at 2025-10-25 20:08:17 on dist-test-slave-d2pk"
server_key: "b47633341d3ee809c5d79736f253e768"
server_key_iv: "8d26280dd8e924e58bf27bd186e627d9"
server_key_version: "encryptionkey@0"
I20251025 20:08:17.191152 22261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "b01df4be952242ee9cf62067c18fa3a9"
format_stamp: "Formatted at 2025-10-25 20:08:17 on dist-test-slave-d2pk"
server_key: "b47633341d3ee809c5d79736f253e768"
server_key_iv: "8d26280dd8e924e58bf27bd186e627d9"
server_key_version: "encryptionkey@0"
I20251025 20:08:17.194018 22261 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:17.195866 22276 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:17.196689 22261 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:17.196772 22261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "b01df4be952242ee9cf62067c18fa3a9"
format_stamp: "Formatted at 2025-10-25 20:08:17 on dist-test-slave-d2pk"
server_key: "b47633341d3ee809c5d79736f253e768"
server_key_iv: "8d26280dd8e924e58bf27bd186e627d9"
server_key_version: "encryptionkey@0"
I20251025 20:08:17.196854 22261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:17.218869 22261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:17.221556 22261 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:17.221702 22261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:17.228121 22328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:43933 every 8 connection(s)
I20251025 20:08:17.228124 22261 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:43933
I20251025 20:08:17.229029 22261 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:17.231660 22329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:17.235522 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22261
I20251025 20:08:17.235615 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:17.235855 18761 external_mini_cluster.cc:1468] Setting key 9e5c191e3714c223effdbd1cd879cd42
I20251025 20:08:17.237036 22329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: Bootstrap starting.
I20251025 20:08:17.239177 22329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:17.239877 22329 log.cc:826] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:17.241540 22329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: No bootstrap required, opened a new log
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:17.244575 22329 raft_consensus.cc:359] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } }
I20251025 20:08:17.244789 22329 raft_consensus.cc:385] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:17.244881 22329 raft_consensus.cc:740] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01df4be952242ee9cf62067c18fa3a9, State: Initialized, Role: FOLLOWER
I20251025 20:08:17.245299 22329 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } }
I20251025 20:08:17.245429 22329 raft_consensus.cc:399] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:17.245494 22329 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:17.245609 22329 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:17.246628 22329 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } }
I20251025 20:08:17.246994 22329 leader_election.cc:304] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b01df4be952242ee9cf62067c18fa3a9; no voters:
I20251025 20:08:17.247274 22329 leader_election.cc:290] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:17.247416 22334 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:17.247603 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:17.236868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.18.82.126:43933)
1025 20:08:17.237212 (+ 344us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:17.237221 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:17.237252 (+ 31us) server_negotiation.cc:408] Connection header received
1025 20:08:17.237747 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:17.237764 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:17.238015 (+ 251us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:17.238241 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:17.239134 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:17.239904 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:17.240550 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:17.240839 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:17.242927 (+ 2088us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:17.242943 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:17.242952 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:17.242980 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:17.244723 (+ 1743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:17.245156 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:17.245162 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:17.245166 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:17.245234 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:17.245453 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:17.245457 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:17.245459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:17.245872 (+ 413us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:17.246059 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:17.246265 (+ 206us) server_negotiation.cc:300] Negotiation successful
1025 20:08:17.246594 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20251025 20:08:17.247634 22334 raft_consensus.cc:697] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [term 1 LEADER]: Becoming Leader. State: Replica: b01df4be952242ee9cf62067c18fa3a9, State: Running, Role: LEADER
I20251025 20:08:17.248296 22334 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } }
I20251025 20:08:17.248410 22329 sys_catalog.cc:565] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:17.249848 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b01df4be952242ee9cf62067c18fa3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } } }
I20251025 20:08:17.249972 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:17.250319 18761 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 20:08:17.250326 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01df4be952242ee9cf62067c18fa3a9. Latest consensus state: current_term: 1 leader_uuid: "b01df4be952242ee9cf62067c18fa3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01df4be952242ee9cf62067c18fa3a9" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43933 } } }
I20251025 20:08:17.250418 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9 [sys.catalog]: This master's current role is: LEADER
W20251025 20:08:17.253422 22349 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:17.253532 22349 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:17.253616 22343 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:17.254400 22343 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:17.259707 22343 catalog_manager.cc:1357] Generated new cluster ID: 6b332cbd44e8439b945625b02b331348
I20251025 20:08:17.259768 22343 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:17.269140 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:17.265649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.18.82.126:43933)
1025 20:08:17.265759 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:17.265763 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:17.265775 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:17.265806 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:17.265808 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:17.265856 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:17.265928 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:17.266570 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:17.267020 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:17.267670 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:17.267885 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:17.268347 (+ 462us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:17.268365 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:17.268367 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:17.268747 (+ 380us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:17.268774 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:17.268788 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:17.268863 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:17.268969 (+ 106us) server_negotiation.cc:300] Negotiation successful
1025 20:08:17.269018 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 20:08:17.293449 22343 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:17.294292 22343 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:17.308557 22343 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b01df4be952242ee9cf62067c18fa3a9: Generated new TSK 0
I20251025 20:08:17.309350 22343 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:08:17.322825 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22261
2025-10-25T20:08:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2211 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22362](info): set up 2 sockets
Oct 25 20:08:17 dist-test-slave-d2pk krb5kdc[22362](info): commencing operation
krb5kdc: starting...
W20251025 20:08:19.166159 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.823s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:19.286077 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:36411
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:19.381990 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:19.382184 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:19.382236 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:19.384965 22378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:19.385012 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:19.385031 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:19.385046 22378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:19.385061 22378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:19.388808 22378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:19.389726 22378 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:19.390509 22378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:19.395490 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:19.395577 22378 server_base.cc:1047] running on GCE node
W20251025 20:08:19.395658 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:19.395743 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:19.396142 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:19.396883 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:19.398015 22378 hybrid_clock.cc:648] HybridClock initialized: now 1761422899398008 us; error 23 us; skew 500 ppm
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:19.400370 22378 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:19.401283 22378 webserver.cc:492] Webserver started at http://127.18.82.126:37663/ using document root <none> and password file <none>
I20251025 20:08:19.401809 22378 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:19.401868 22378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:19.402029 22378 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:19.403498 22378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "8c4a25abbf304c97988ea7aa38969d85"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "bbdffb55d633d011dd96426e2b8d8589"
server_key_iv: "8c3d9e4bc28c79450ec3682ba019cb83"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.403964 22378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "8c4a25abbf304c97988ea7aa38969d85"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "bbdffb55d633d011dd96426e2b8d8589"
server_key_iv: "8c3d9e4bc28c79450ec3682ba019cb83"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.407244 22378 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 20:08:19.409178 22393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.410127 22378 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:19.410235 22378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "8c4a25abbf304c97988ea7aa38969d85"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "bbdffb55d633d011dd96426e2b8d8589"
server_key_iv: "8c3d9e4bc28c79450ec3682ba019cb83"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.410320 22378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:19.429370 22378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:19.435708 22378 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:19.435860 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:19.442505 22378 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:36411
I20251025 20:08:19.442520 22445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:36411 every 8 connection(s)
I20251025 20:08:19.443409 22378 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:19.446111 22446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:19.450953 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22378
I20251025 20:08:19.451071 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:19.451287 18761 external_mini_cluster.cc:1468] Setting key 91f5d17ffc19fa3bf7bc684401a7afa3
I20251025 20:08:19.451432 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85: Bootstrap starting.
I20251025 20:08:19.453605 22446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:19.454475 22446 log.cc:826] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:19.456192 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85: No bootstrap required, opened a new log
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:19.459014 22446 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } }
I20251025 20:08:19.459170 22446 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:19.459218 22446 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4a25abbf304c97988ea7aa38969d85, State: Initialized, Role: FOLLOWER
I20251025 20:08:19.459661 22446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } }
I20251025 20:08:19.459810 22446 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:19.459885 22446 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:19.459985 22446 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:19.460789 22446 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } }
I20251025 20:08:19.461117 22446 leader_election.cc:304] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4a25abbf304c97988ea7aa38969d85; no voters:
I20251025 20:08:19.461438 22446 leader_election.cc:290] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:19.461519 22451 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:19.461750 22451 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4a25abbf304c97988ea7aa38969d85, State: Running, Role: LEADER
I20251025 20:08:19.462067 22451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } }
I20251025 20:08:19.462641 22446 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:19.463640 22453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4a25abbf304c97988ea7aa38969d85. Latest consensus state: current_term: 1 leader_uuid: "8c4a25abbf304c97988ea7aa38969d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } } }
I20251025 20:08:19.463769 22453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:19.464068 22452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4a25abbf304c97988ea7aa38969d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a25abbf304c97988ea7aa38969d85" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 36411 } } }
I20251025 20:08:19.464143 22460 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:19.464148 22452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:19.464231 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:19.452360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.18.82.126:36411)
1025 20:08:19.452712 (+ 352us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:19.452721 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:19.452764 (+ 43us) server_negotiation.cc:408] Connection header received
1025 20:08:19.453197 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:19.453211 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:19.453476 (+ 265us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:19.453781 (+ 305us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:19.454766 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.455462 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:19.456181 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.456435 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:19.458691 (+ 2256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:19.458706 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:19.458716 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:19.458744 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:19.460544 (+ 1800us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.460991 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.460996 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.460999 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.461060 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.461302 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.461305 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.461307 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.461783 (+ 476us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:19.462993 (+ 1210us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:19.463161 (+ 168us) server_negotiation.cc:300] Negotiation successful
1025 20:08:19.463422 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20251025 20:08:19.467120 22460 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:19.472224 22460 catalog_manager.cc:1357] Generated new cluster ID: 0c10238305894ded8422fa25bf751888
I20251025 20:08:19.472275 22460 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:19.482877 22460 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:19.483870 22460 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:19.491974 22460 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8c4a25abbf304c97988ea7aa38969d85: Generated new TSK 0
I20251025 20:08:19.492538 22460 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:19.545923 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36411
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:19.651624 22474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:19.651868 22474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:19.651969 22474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:19.656322 22474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:19.656385 22474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:19.656476 22474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:19.660466 22474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:19.661397 22474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:19.662209 22474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:19.668592 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:19.668603 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:19.668603 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:19.668758 22474 server_base.cc:1047] running on GCE node
I20251025 20:08:19.669152 22474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:19.669695 22474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:19.670828 22474 hybrid_clock.cc:648] HybridClock initialized: now 1761422899670817 us; error 29 us; skew 500 ppm
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:19.673117 22474 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:19.674038 22474 webserver.cc:492] Webserver started at http://127.18.82.65:43233/ using document root <none> and password file <none>
I20251025 20:08:19.674510 22474 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:19.674553 22474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:19.674719 22474 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:19.676178 22474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "3199860df04e4a34b126c66f74385655"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "6b59b304ff730dd34738e5c92f805cf9"
server_key_iv: "c17d57989f4dce6cbc383eb375d0c218"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.676565 22474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "3199860df04e4a34b126c66f74385655"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "6b59b304ff730dd34738e5c92f805cf9"
server_key_iv: "c17d57989f4dce6cbc383eb375d0c218"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.679579 22474 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 20:08:19.681421 22489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.682274 22474 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:19.682356 22474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "3199860df04e4a34b126c66f74385655"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "6b59b304ff730dd34738e5c92f805cf9"
server_key_iv: "c17d57989f4dce6cbc383eb375d0c218"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.682439 22474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:19.695883 22474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:19.698246 22474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:19.698467 22474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:19.698958 22474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:19.699764 22474 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:19.699808 22474 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.699853 22474 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:19.699887 22474 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.707657 22474 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:35237
I20251025 20:08:19.707686 22602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:35237 every 8 connection(s)
I20251025 20:08:19.708458 22474 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:19.717705 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22474
I20251025 20:08:19.717804 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:19.718022 18761 external_mini_cluster.cc:1468] Setting key 4173992ed55927f96d12cfe305aa76d3
I20251025 20:08:19.719290 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:19.710009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:49055 (local address 127.18.82.126:36411)
1025 20:08:19.710141 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:19.710145 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:19.710767 (+ 622us) server_negotiation.cc:408] Connection header received
1025 20:08:19.711511 (+ 744us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:19.711515 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:19.711576 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:19.711646 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:19.712746 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.713187 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:19.713792 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.713961 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:19.716240 (+ 2279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:19.716259 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:19.716262 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:19.716285 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:19.717395 (+ 1110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.717871 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.717874 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.717876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.717924 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.718199 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.718203 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.718204 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.718361 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:19.718485 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:19.719016 (+ 531us) server_negotiation.cc:300] Negotiation successful
1025 20:08:19.719140 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:08:19.719924 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:19.710217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36411 (local address 127.18.82.65:49055)
1025 20:08:19.710607 (+ 390us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:19.710661 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:19.711326 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:19.711797 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:19.711805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:19.712119 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:19.712596 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:19.712606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.713291 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:19.713294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:19.713669 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:19.713676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.713877 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:19.714852 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:19.714872 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:19.716103 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:19.717490 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:19.717494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:19.717502 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:19.717747 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:19.718024 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:19.718026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:19.718027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:19.718113 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:19.718508 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:19.718514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:19.718765 (+ 251us) client_negotiation.cc:769] Sending connection context
1025 20:08:19.718957 (+ 192us) client_negotiation.cc:240] Negotiation successful
1025 20:08:19.719170 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20251025 20:08:19.721131 22603 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36411
I20251025 20:08:19.721338 22603 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:19.721808 22603 heartbeater.cc:507] Master 127.18.82.126:36411 requested a full tablet report, sending...
I20251025 20:08:19.723201 22410 ts_manager.cc:194] Registered new tserver with Master: 3199860df04e4a34b126c66f74385655 (127.18.82.65:35237)
I20251025 20:08:19.724762 22410 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:49055
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:19.760078 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36411
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:19.855482 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:19.855648 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:19.855695 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:19.858449 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:19.858495 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:19.858565 22610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:19.862439 22610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:19.863252 22610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:19.864032 22610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:19.869987 22616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:19.869982 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:19.870005 22615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:19.870076 22610 server_base.cc:1047] running on GCE node
I20251025 20:08:19.870540 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:19.871021 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:19.872162 22610 hybrid_clock.cc:648] HybridClock initialized: now 1761422899872143 us; error 33 us; skew 500 ppm
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:19.874528 22610 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:19.875361 22610 webserver.cc:492] Webserver started at http://127.18.82.66:36905/ using document root <none> and password file <none>
I20251025 20:08:19.875792 22610 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:19.875830 22610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:19.875967 22610 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:19.877418 22610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "33632dcfb15a4a558613f1bc7ff467b5"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "852091c345a1372a05f511a62822ee34"
server_key_iv: "23d59642fb42d2a84d9c8528ecc2f5e9"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.877854 22610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "33632dcfb15a4a558613f1bc7ff467b5"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "852091c345a1372a05f511a62822ee34"
server_key_iv: "23d59642fb42d2a84d9c8528ecc2f5e9"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.880767 22610 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:19.882563 22625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.883459 22610 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:19.883554 22610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "33632dcfb15a4a558613f1bc7ff467b5"
format_stamp: "Formatted at 2025-10-25 20:08:19 on dist-test-slave-d2pk"
server_key: "852091c345a1372a05f511a62822ee34"
server_key_iv: "23d59642fb42d2a84d9c8528ecc2f5e9"
server_key_version: "encryptionkey@0"
I20251025 20:08:19.883638 22610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:19.893469 22610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:19.895756 22610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:19.895891 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:19.896327 22610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:19.897079 22610 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:19.897125 22610 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.897178 22610 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:19.897208 22610 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:19.905251 22610 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:44711
I20251025 20:08:19.905277 22738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:44711 every 8 connection(s)
I20251025 20:08:19.906152 22610 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
Oct 25 20:08:19 dist-test-slave-d2pk krb5kdc[22362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422899, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:19.914695 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22610
I20251025 20:08:19.914805 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:19.915073 18761 external_mini_cluster.cc:1468] Setting key af0abbe96f8b1d002fdf3b8c0208c41e
I20251025 20:08:19.917840 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:19.907883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:44227 (local address 127.18.82.126:36411)
1025 20:08:19.908021 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:19.908026 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:19.908614 (+ 588us) server_negotiation.cc:408] Connection header received
1025 20:08:19.909460 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:19.909464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:19.909571 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:19.909647 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:19.910888 (+ 1241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.911322 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:19.911878 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.912057 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:19.914547 (+ 2490us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:19.914564 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:19.914567 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:19.914590 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:19.916027 (+ 1437us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.916504 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.916508 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.916510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.916559 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:19.916787 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:19.916792 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:19.916795 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:19.916975 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:19.917075 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:19.917477 (+ 402us) server_negotiation.cc:300] Negotiation successful
1025 20:08:19.917615 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:08:19.918329 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:19.908106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36411 (local address 127.18.82.66:44227)
1025 20:08:19.908482 (+ 376us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:19.908515 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:19.909257 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:19.909824 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:19.909832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:19.910193 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:19.910735 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:19.910747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.911425 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:19.911430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:19.911770 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:19.911776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:19.911980 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:19.913047 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:19.913068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:19.914408 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:19.916138 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:19.916142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:19.916150 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:19.916419 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:19.916642 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:19.916644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:19.916645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:19.916715 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:19.917061 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:19.917065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:19.917258 (+ 193us) client_negotiation.cc:769] Sending connection context
1025 20:08:19.917419 (+ 161us) client_negotiation.cc:240] Negotiation successful
1025 20:08:19.917665 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20251025 20:08:19.919572 22739 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36411
I20251025 20:08:19.919791 22739 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:19.920241 22739 heartbeater.cc:507] Master 127.18.82.126:36411 requested a full tablet report, sending...
I20251025 20:08:19.921279 22410 ts_manager.cc:194] Registered new tserver with Master: 33632dcfb15a4a558613f1bc7ff467b5 (127.18.82.66:44711)
I20251025 20:08:19.922098 22410 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:44227
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:19.960435 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:36411
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:20.059862 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:20.060055 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:20.060107 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:20.063191 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:20.063246 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:20.063320 22746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:20.067378 22746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:20.068282 22746 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:20.069128 22746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:20.075232 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:20.075328 22746 server_base.cc:1047] running on GCE node
W20251025 20:08:20.075253 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:20.075219 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:20.075774 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:20.076256 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:20.077404 22746 hybrid_clock.cc:648] HybridClock initialized: now 1761422900077395 us; error 30 us; skew 500 ppm
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422900, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:20.079962 22746 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:20.081040 22746 webserver.cc:492] Webserver started at http://127.18.82.67:39663/ using document root <none> and password file <none>
I20251025 20:08:20.081652 22746 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:20.081705 22746 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:20.081897 22746 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:20.083527 22746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "056d2fec45084f8e9b1c32bfcf4a03b0"
format_stamp: "Formatted at 2025-10-25 20:08:20 on dist-test-slave-d2pk"
server_key: "c9ee38499d9b26999c9d39b095bac36f"
server_key_iv: "75e90c45340b5d4537631a4ff325e4ba"
server_key_version: "encryptionkey@0"
I20251025 20:08:20.083990 22746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "056d2fec45084f8e9b1c32bfcf4a03b0"
format_stamp: "Formatted at 2025-10-25 20:08:20 on dist-test-slave-d2pk"
server_key: "c9ee38499d9b26999c9d39b095bac36f"
server_key_iv: "75e90c45340b5d4537631a4ff325e4ba"
server_key_version: "encryptionkey@0"
I20251025 20:08:20.087390 22746 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:20.089407 22761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:20.090394 22746 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:20.090498 22746 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "056d2fec45084f8e9b1c32bfcf4a03b0"
format_stamp: "Formatted at 2025-10-25 20:08:20 on dist-test-slave-d2pk"
server_key: "c9ee38499d9b26999c9d39b095bac36f"
server_key_iv: "75e90c45340b5d4537631a4ff325e4ba"
server_key_version: "encryptionkey@0"
I20251025 20:08:20.090582 22746 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:20.107158 22746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:20.109634 22746 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:20.109800 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:20.110316 22746 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:20.111162 22746 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:20.111207 22746 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:20.111260 22746 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:20.111290 22746 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:20.119169 22746 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:41927
I20251025 20:08:20.119196 22874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:41927 every 8 connection(s)
I20251025 20:08:20.120085 22746 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:20.125361 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22746
I20251025 20:08:20.125454 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:20.125689 18761 external_mini_cluster.cc:1468] Setting key e3c41263b7b10cb3b6b7139abf90e945
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422900, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:20.131429 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:20.121708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:53513 (local address 127.18.82.126:36411)
1025 20:08:20.121836 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:20.121839 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:20.122480 (+ 641us) server_negotiation.cc:408] Connection header received
1025 20:08:20.123228 (+ 748us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:20.123232 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:20.123292 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:20.123362 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:20.124573 (+ 1211us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.125067 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:20.125707 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.125879 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:20.128676 (+ 2797us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:20.128691 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:20.128693 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:20.128719 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:20.130005 (+ 1286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:20.130422 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:20.130424 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:20.130425 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:20.130460 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:20.130677 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:20.130679 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:20.130681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:20.130819 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:20.130886 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:20.131247 (+ 361us) server_negotiation.cc:300] Negotiation successful
1025 20:08:20.131326 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:08:20.132006 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:20.121942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:36411 (local address 127.18.82.67:53513)
1025 20:08:20.122323 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:20.122370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:20.123039 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:20.123530 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:20.123537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:20.123911 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:20.124418 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:20.124430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.125179 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:20.125182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:20.125593 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:20.125601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.125793 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:20.126955 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:20.126977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:20.128536 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:20.130108 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:20.130112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:20.130121 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:20.130340 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:20.130519 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:20.130521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:20.130522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:20.130589 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:20.130885 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:20.130888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:20.131078 (+ 190us) client_negotiation.cc:769] Sending connection context
1025 20:08:20.131224 (+ 146us) client_negotiation.cc:240] Negotiation successful
1025 20:08:20.131402 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20251025 20:08:20.132867 22875 heartbeater.cc:344] Connected to a master server at 127.18.82.126:36411
I20251025 20:08:20.133080 22875 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:20.133558 22875 heartbeater.cc:507] Master 127.18.82.126:36411 requested a full tablet report, sending...
I20251025 20:08:20.134485 22410 ts_manager.cc:194] Registered new tserver with Master: 056d2fec45084f8e9b1c32bfcf4a03b0 (127.18.82.67:41927)
I20251025 20:08:20.134972 22410 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:53513
I20251025 20:08:20.139163 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422900, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422900, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:20.160544 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:20.153560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.18.82.126:36411)
1025 20:08:20.153712 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:20.153716 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:20.153811 (+ 95us) server_negotiation.cc:408] Connection header received
1025 20:08:20.153936 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:20.153939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:20.153985 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:20.154048 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:20.154801 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.155264 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:20.155846 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:20.156055 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:20.158012 (+ 1957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:20.158027 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:20.158029 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:20.158052 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:20.159164 (+ 1112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:20.159515 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:20.159521 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:20.159523 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:20.159582 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:20.159801 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:20.159804 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:20.159806 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:20.159964 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:20.160086 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:20.160284 (+ 198us) server_negotiation.cc:300] Negotiation successful
1025 20:08:20.160395 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20251025 20:08:20.160916 22410 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56388
I20251025 20:08:20.161636 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22474
I20251025 20:08:20.168082 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22610
I20251025 20:08:20.173172 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22746
I20251025 20:08:20.177986 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22378
2025-10-25T20:08:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2856 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22887](info): set up 2 sockets
Oct 25 20:08:20 dist-test-slave-d2pk krb5kdc[22887](info): commencing operation
krb5kdc: starting...
W20251025 20:08:22.038131 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.839s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:22.163719 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:42849
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:22.262845 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:22.263043 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:22.263116 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:22.266103 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:22.266153 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:22.266187 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:22.266224 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:22.266263 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:22.270176 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:42849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:22.271142 22903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:22.271910 22903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:22.276846 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.276854 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.276894 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:22.277086 22903 server_base.cc:1047] running on GCE node
I20251025 20:08:22.277531 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:22.278330 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:22.279486 22903 hybrid_clock.cc:648] HybridClock initialized: now 1761422902279473 us; error 30 us; skew 500 ppm
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:22.281847 22903 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:22.282766 22903 webserver.cc:492] Webserver started at http://127.18.82.126:38523/ using document root <none> and password file <none>
I20251025 20:08:22.283273 22903 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:22.283324 22903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:22.283496 22903 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:22.285064 22903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "5c68cfca8aaa4d4792f5edec40b4aa28"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "239be06ac47d0e43465bd8094bda4323"
server_key_iv: "5e36b78c9fcfb9e041cbfc3a312128d3"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.285569 22903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "5c68cfca8aaa4d4792f5edec40b4aa28"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "239be06ac47d0e43465bd8094bda4323"
server_key_iv: "5e36b78c9fcfb9e041cbfc3a312128d3"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.288703 22903 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:22.290786 22918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.291725 22903 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:22.291841 22903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "5c68cfca8aaa4d4792f5edec40b4aa28"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "239be06ac47d0e43465bd8094bda4323"
server_key_iv: "5e36b78c9fcfb9e041cbfc3a312128d3"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.291922 22903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:22.308633 22903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:22.311089 22903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:22.311239 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:22.317862 22903 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:42849
I20251025 20:08:22.317860 22970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:42849 every 8 connection(s)
I20251025 20:08:22.318760 22903 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:22.321400 22971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:22.326658 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28: Bootstrap starting.
I20251025 20:08:22.328665 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22903
I20251025 20:08:22.328795 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:22.329085 18761 external_mini_cluster.cc:1468] Setting key 09b1ca40ee5724696c71f22361f06909
I20251025 20:08:22.329079 22971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:22.329890 22971 log.cc:826] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:22.331732 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28: No bootstrap required, opened a new log
I20251025 20:08:22.334615 22971 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } }
I20251025 20:08:22.334841 22971 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:22.334911 22971 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c68cfca8aaa4d4792f5edec40b4aa28, State: Initialized, Role: FOLLOWER
I20251025 20:08:22.335440 22971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } }
I20251025 20:08:22.335597 22971 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:22.335678 22971 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:22.335775 22971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 0 FOLLOWER]: Advancing to term 1
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:22.336540 22971 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } }
I20251025 20:08:22.336863 22971 leader_election.cc:304] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c68cfca8aaa4d4792f5edec40b4aa28; no voters:
I20251025 20:08:22.337116 22971 leader_election.cc:290] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:22.337985 22976 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:22.338193 22976 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [term 1 LEADER]: Becoming Leader. State: Replica: 5c68cfca8aaa4d4792f5edec40b4aa28, State: Running, Role: LEADER
I20251025 20:08:22.338250 22971 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:22.338564 22976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } }
I20251025 20:08:22.340170 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } } }
I20251025 20:08:22.340279 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:22.340669 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c68cfca8aaa4d4792f5edec40b4aa28. Latest consensus state: current_term: 1 leader_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68cfca8aaa4d4792f5edec40b4aa28" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 42849 } } }
I20251025 20:08:22.340806 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:22.341711 22988 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:22.342229 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:22.330195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.18.82.126:42849)
1025 20:08:22.330578 (+ 383us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:22.330588 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:22.330621 (+ 33us) server_negotiation.cc:408] Connection header received
1025 20:08:22.331135 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:22.331150 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:22.331421 (+ 271us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:22.331672 (+ 251us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:22.332672 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.333374 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:22.334208 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.334408 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:22.336682 (+ 2274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:22.336702 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:22.336715 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:22.336746 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:22.338979 (+ 2233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.339535 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.339544 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.339553 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.339758 (+ 205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.340086 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.340090 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.340093 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.340577 (+ 484us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:22.340786 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:22.341213 (+ 427us) server_negotiation.cc:300] Negotiation successful
1025 20:08:22.341497 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20251025 20:08:22.344213 22988 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:22.349334 22988 catalog_manager.cc:1357] Generated new cluster ID: 4ce234a330304ed18759f5d9e90c8efb
I20251025 20:08:22.349393 22988 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:22.360908 22988 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:22.362098 22988 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:22.382822 22988 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5c68cfca8aaa4d4792f5edec40b4aa28: Generated new TSK 0
I20251025 20:08:22.383589 22988 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:22.430371 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42849
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:22.533390 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:22.533618 22999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:22.533677 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:22.536496 22999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:22.536545 22999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:22.536612 22999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:22.540381 22999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:22.541368 22999 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:22.542193 22999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:22.548365 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.548379 23007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.548386 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:22.548607 22999 server_base.cc:1047] running on GCE node
I20251025 20:08:22.548914 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:22.549408 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:22.550554 22999 hybrid_clock.cc:648] HybridClock initialized: now 1761422902550546 us; error 25 us; skew 500 ppm
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:22.552883 22999 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:22.553879 22999 webserver.cc:492] Webserver started at http://127.18.82.65:43341/ using document root <none> and password file <none>
I20251025 20:08:22.554394 22999 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:22.554446 22999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:22.554627 22999 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:22.556293 22999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "d520e25e5fe440a69d0bbbfd58da8809"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "8abf0c2d964983a040b4b060467d893b"
server_key_iv: "9195b4acb69ee902b0a7c1aa65bad750"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.556787 22999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "d520e25e5fe440a69d0bbbfd58da8809"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "8abf0c2d964983a040b4b060467d893b"
server_key_iv: "9195b4acb69ee902b0a7c1aa65bad750"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.560135 22999 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:22.562175 23014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.563185 22999 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:22.563273 22999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "d520e25e5fe440a69d0bbbfd58da8809"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "8abf0c2d964983a040b4b060467d893b"
server_key_iv: "9195b4acb69ee902b0a7c1aa65bad750"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.563366 22999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:22.575861 22999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:22.578190 22999 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:22.578348 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:22.578848 22999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:22.579679 22999 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:22.579727 22999 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.579769 22999 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:22.579802 22999 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.588305 22999 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:33295
I20251025 20:08:22.588332 23127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:33295 every 8 connection(s)
I20251025 20:08:22.589157 22999 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:22.595439 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 22999
I20251025 20:08:22.595548 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:22.595813 18761 external_mini_cluster.cc:1468] Setting key a0952607bc63a98a6a9e9a4a6c57a311
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:22.601593 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:22.590775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:36877 (local address 127.18.82.126:42849)
1025 20:08:22.590904 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:22.590908 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:22.591542 (+ 634us) server_negotiation.cc:408] Connection header received
1025 20:08:22.592331 (+ 789us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:22.592335 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:22.592394 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:22.592466 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:22.594032 (+ 1566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.594512 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:22.595127 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.595311 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:22.597999 (+ 2688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:22.598039 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:22.598046 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:22.598081 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:22.599612 (+ 1531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.600071 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.600078 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.600082 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.600139 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.600367 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.600372 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.600373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.600586 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:22.600708 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:22.601192 (+ 484us) server_negotiation.cc:300] Negotiation successful
1025 20:08:22.601362 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 20:08:22.602115 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:22.591026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42849 (local address 127.18.82.65:36877)
1025 20:08:22.591415 (+ 389us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:22.591448 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:22.592153 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:22.592639 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:22.592647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:22.592991 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:22.593893 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:22.593907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.594630 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:22.594633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:22.595014 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:22.595021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.595247 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:22.595959 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:22.596011 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:22.597847 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:22.599727 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:22.599731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:22.599739 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:22.599978 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:22.600219 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:22.600221 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:22.600223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:22.600294 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:22.600724 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:22.600732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:22.600946 (+ 214us) client_negotiation.cc:769] Sending connection context
1025 20:08:22.601119 (+ 173us) client_negotiation.cc:240] Negotiation successful
1025 20:08:22.601345 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
I20251025 20:08:22.603226 23128 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42849
I20251025 20:08:22.603439 23128 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:22.603880 23128 heartbeater.cc:507] Master 127.18.82.126:42849 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
I20251025 20:08:22.605247 22935 ts_manager.cc:194] Registered new tserver with Master: d520e25e5fe440a69d0bbbfd58da8809 (127.18.82.65:33295)
I20251025 20:08:22.606434 22935 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:36877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:22.639649 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42849
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:22.741954 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:22.742144 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:22.742223 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:22.745123 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:22.745177 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:22.745275 23135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:22.749086 23135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:22.750017 23135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:22.750802 23135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:22.756727 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.756717 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.756707 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:22.756973 23135 server_base.cc:1047] running on GCE node
I20251025 20:08:22.757278 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:22.757817 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:22.758955 23135 hybrid_clock.cc:648] HybridClock initialized: now 1761422902758943 us; error 26 us; skew 500 ppm
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:22.761405 23135 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:22.762356 23135 webserver.cc:492] Webserver started at http://127.18.82.66:44779/ using document root <none> and password file <none>
I20251025 20:08:22.762828 23135 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:22.762871 23135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:22.763010 23135 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:22.764603 23135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "5315a290f7b34f299869f422521d287d"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "9e4b2951eb53ed0a5fb72d26dc88922e"
server_key_iv: "f2470e1b791d623915cb11044c9c3c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.765031 23135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "5315a290f7b34f299869f422521d287d"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "9e4b2951eb53ed0a5fb72d26dc88922e"
server_key_iv: "f2470e1b791d623915cb11044c9c3c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.768184 23135 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:22.770274 23150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.771235 23135 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251025 20:08:22.771323 23135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "5315a290f7b34f299869f422521d287d"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "9e4b2951eb53ed0a5fb72d26dc88922e"
server_key_iv: "f2470e1b791d623915cb11044c9c3c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.771428 23135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:22.781234 23135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:22.783712 23135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:22.783865 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:22.784379 23135 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:22.785183 23135 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:22.785223 23135 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.785274 23135 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:22.785302 23135 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.793769 23135 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:37753
I20251025 20:08:22.793790 23263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:37753 every 8 connection(s)
I20251025 20:08:22.794642 23135 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:22.804554 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23135
I20251025 20:08:22.804687 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:22.804980 18761 external_mini_cluster.cc:1468] Setting key b461037bc179c720759d070cf6a2b804
I20251025 20:08:22.806108 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:22.796292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:51171 (local address 127.18.82.126:42849)
1025 20:08:22.796436 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:22.796440 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:22.797120 (+ 680us) server_negotiation.cc:408] Connection header received
1025 20:08:22.798018 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:22.798022 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:22.798085 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:22.798155 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:22.799671 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.800176 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:22.800819 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.801032 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:22.803004 (+ 1972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:22.803023 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:22.803025 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:22.803050 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:22.804194 (+ 1144us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.804749 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.804753 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.804756 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.804818 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:22.805107 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:22.805112 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:22.805114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:22.805248 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:22.805335 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:22.805849 (+ 514us) server_negotiation.cc:300] Negotiation successful
1025 20:08:22.805968 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:22.806833 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:22.796546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42849 (local address 127.18.82.66:51171)
1025 20:08:22.796977 (+ 431us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:22.797010 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:22.797829 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:22.798313 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:22.798320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:22.798672 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:22.799518 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:22.799530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.800295 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:22.800298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:22.800699 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:22.800707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:22.800911 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:22.801468 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:22.801490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:22.802803 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:22.804303 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:22.804308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:22.804317 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:22.804637 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:22.804901 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:22.804904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:22.804906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:22.805016 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:22.805352 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:22.805358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:22.805630 (+ 272us) client_negotiation.cc:769] Sending connection context
1025 20:08:22.805800 (+ 170us) client_negotiation.cc:240] Negotiation successful
1025 20:08:22.806024 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":133,"threads_started":1}
I20251025 20:08:22.808008 23264 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42849
I20251025 20:08:22.808249 23264 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:22.808785 23264 heartbeater.cc:507] Master 127.18.82.126:42849 requested a full tablet report, sending...
I20251025 20:08:22.809779 22935 ts_manager.cc:194] Registered new tserver with Master: 5315a290f7b34f299869f422521d287d (127.18.82.66:37753)
I20251025 20:08:22.810593 22935 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:51171
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:22.846298 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:42849
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:22.947398 23271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:22.947639 23271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:22.947721 23271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:22.950834 23271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:22.950891 23271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:22.950999 23271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:22.954838 23271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:22.955850 23271 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:22.956655 23271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:22.962908 23276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.962931 23279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:22.962908 23277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:22.963117 23271 server_base.cc:1047] running on GCE node
I20251025 20:08:22.963464 23271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:22.963953 23271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:22.965091 23271 hybrid_clock.cc:648] HybridClock initialized: now 1761422902965070 us; error 35 us; skew 500 ppm
Oct 25 20:08:22 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:22.967551 23271 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:22.968513 23271 webserver.cc:492] Webserver started at http://127.18.82.67:42807/ using document root <none> and password file <none>
I20251025 20:08:22.968993 23271 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:22.969038 23271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:22.969178 23271 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:22.970808 23271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "dbcccc3871694da2a13178689f212c4f"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "fb9019fc54f4b70fab4d97a22e2a6886"
server_key_iv: "6e4fb2cc9f7595621320782a1e5a4db7"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.971242 23271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "dbcccc3871694da2a13178689f212c4f"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "fb9019fc54f4b70fab4d97a22e2a6886"
server_key_iv: "6e4fb2cc9f7595621320782a1e5a4db7"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.974305 23271 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:22.976294 23286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.977174 23271 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:22.977255 23271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "dbcccc3871694da2a13178689f212c4f"
format_stamp: "Formatted at 2025-10-25 20:08:22 on dist-test-slave-d2pk"
server_key: "fb9019fc54f4b70fab4d97a22e2a6886"
server_key_iv: "6e4fb2cc9f7595621320782a1e5a4db7"
server_key_version: "encryptionkey@0"
I20251025 20:08:22.977342 23271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:22.992400 23271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:22.994737 23271 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:22.994865 23271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:22.995289 23271 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:22.996071 23271 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:22.996109 23271 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:22.996145 23271 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:22.996163 23271 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:23.004637 23271 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:43435
I20251025 20:08:23.004662 23399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:43435 every 8 connection(s)
I20251025 20:08:23.005508 23271 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:23.011674 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23271
I20251025 20:08:23.012110 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:23.012356 18761 external_mini_cluster.cc:1468] Setting key d1ba33d67ede9d258167bd88040042ac
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:23.016774 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:23.007147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:32967 (local address 127.18.82.126:42849)
1025 20:08:23.007297 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:23.007302 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:23.007964 (+ 662us) server_negotiation.cc:408] Connection header received
1025 20:08:23.008669 (+ 705us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:23.008673 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:23.008722 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:23.008790 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:23.010418 (+ 1628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.010914 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:23.011549 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.011792 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:23.013954 (+ 2162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:23.013966 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:23.013969 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:23.013992 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:23.015385 (+ 1393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.015779 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.015782 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.015783 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.015817 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.016005 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.016006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.016008 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.016129 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:23.016195 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:23.016577 (+ 382us) server_negotiation.cc:300] Negotiation successful
1025 20:08:23.016665 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:23.017335 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:23.007415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:42849 (local address 127.18.82.67:32967)
1025 20:08:23.007828 (+ 413us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:23.007860 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:23.008498 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:23.008916 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:23.008923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:23.009275 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:23.010272 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:23.010286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.011016 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:23.011019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:23.011411 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:23.011418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.011573 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:23.012137 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:23.012169 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:23.013817 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:23.015486 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:23.015490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:23.015498 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:23.015704 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:23.015875 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:23.015877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:23.015879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:23.015945 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:23.016196 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:23.016200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:23.016398 (+ 198us) client_negotiation.cc:769] Sending connection context
1025 20:08:23.016551 (+ 153us) client_negotiation.cc:240] Negotiation successful
1025 20:08:23.016734 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20251025 20:08:23.018297 23400 heartbeater.cc:344] Connected to a master server at 127.18.82.126:42849
I20251025 20:08:23.018519 23400 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:23.018970 23400 heartbeater.cc:507] Master 127.18.82.126:42849 requested a full tablet report, sending...
I20251025 20:08:23.019903 22935 ts_manager.cc:194] Registered new tserver with Master: dbcccc3871694da2a13178689f212c4f (127.18.82.67:43435)
I20251025 20:08:23.020426 22935 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:32967
I20251025 20:08:23.026105 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:08:23.047412 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:23.039894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.18.82.65:33295)
1025 20:08:23.040221 (+ 327us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:23.040227 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:23.040249 (+ 22us) server_negotiation.cc:408] Connection header received
1025 20:08:23.040300 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:23.040305 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:23.040468 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:23.040585 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:23.041331 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.042064 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:23.042610 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.042774 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:23.044591 (+ 1817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:23.044620 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:23.044631 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:23.044657 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:23.046123 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.046490 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.046495 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.046496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.046546 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.046727 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.046729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.046731 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.046895 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:23.047035 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:23.047170 (+ 135us) server_negotiation.cc:300] Negotiation successful
1025 20:08:23.047282 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[22887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:23.056414 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:23.049862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.18.82.126:42849)
1025 20:08:23.050025 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:23.050029 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:23.050044 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:23.050146 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:23.050149 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:23.050196 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:23.050264 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:23.050938 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.051417 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:23.051973 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:23.052198 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:23.054099 (+ 1901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:23.054115 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:23.054117 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:23.054141 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:23.055270 (+ 1129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.055588 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.055590 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.055591 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.055631 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:23.055797 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:23.055799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:23.055800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:23.055934 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:23.056044 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:23.056187 (+ 143us) server_negotiation.cc:300] Negotiation successful
1025 20:08:23.056269 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20251025 20:08:23.056790 22935 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45634
I20251025 20:08:23.057340 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22999
I20251025 20:08:23.063905 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23135
I20251025 20:08:23.071503 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23271
I20251025 20:08:23.076771 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 22903
2025-10-25T20:08:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2898 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[23412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[23412](info): set up 2 sockets
Oct 25 20:08:23 dist-test-slave-d2pk krb5kdc[23412](info): commencing operation
krb5kdc: starting...
W20251025 20:08:24.942426 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.844s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:25.064538 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:37943
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:37943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:25.162037 23428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:25.162223 23428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:25.162272 23428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:25.165335 23428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:25.165397 23428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:25.165418 23428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:25.165441 23428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:25.165457 23428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:25.169299 23428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:37943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:37943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:25.170267 23428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:25.171046 23428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:25.176116 23436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.176133 23433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.176110 23434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.176577 23428 server_base.cc:1047] running on GCE node
I20251025 20:08:25.177026 23428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:25.177918 23428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:25.179062 23428 hybrid_clock.cc:648] HybridClock initialized: now 1761422905179036 us; error 39 us; skew 500 ppm
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:25.181591 23428 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:25.182289 23428 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:25.182340 23428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:25.182505 23428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:25.184073 23428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "5bab2cf528ee48f6a9febcdb5d79b16b"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "fd9712ef42e2d48058f4652e748a20ad"
server_key_iv: "96f383ce7b90686b1b4c1b1226f60d2a"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.184514 23428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "5bab2cf528ee48f6a9febcdb5d79b16b"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "fd9712ef42e2d48058f4652e748a20ad"
server_key_iv: "96f383ce7b90686b1b4c1b1226f60d2a"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.187700 23428 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:25.189708 23442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.190660 23428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:25.190755 23428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "5bab2cf528ee48f6a9febcdb5d79b16b"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "fd9712ef42e2d48058f4652e748a20ad"
server_key_iv: "96f383ce7b90686b1b4c1b1226f60d2a"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.190840 23428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:25.203881 23428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:25.208988 23428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:25.209138 23428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:25.215865 23494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:37943 every 8 connection(s)
I20251025 20:08:25.215888 23428 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:37943
I20251025 20:08:25.216427 23428 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:25.218966 23495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:25.219331 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23428
I20251025 20:08:25.219444 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:25.219696 18761 external_mini_cluster.cc:1468] Setting key d7bd38c568c8feaa72de4f045ea00a87
I20251025 20:08:25.224694 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: Bootstrap starting.
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:25.226950 23495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:25.227633 23495 log.cc:826] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:25.229103 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: No bootstrap required, opened a new log
I20251025 20:08:25.230506 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.220963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.18.82.126:37943)
1025 20:08:25.221355 (+ 392us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.221364 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.221398 (+ 34us) server_negotiation.cc:408] Connection header received
1025 20:08:25.221932 (+ 534us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.221947 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.222197 (+ 250us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.222469 (+ 272us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:25.223342 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.224038 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.224634 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.224885 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.226674 (+ 1789us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:25.226692 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:25.226701 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:25.226732 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:25.228310 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.228675 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.228680 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.228683 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.228745 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.228949 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.228952 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.228953 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.229276 (+ 323us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:25.229463 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.229740 (+ 277us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.229955 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20251025 20:08:25.231626 23495 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } }
I20251025 20:08:25.231806 23495 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:25.231858 23495 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bab2cf528ee48f6a9febcdb5d79b16b, State: Initialized, Role: FOLLOWER
I20251025 20:08:25.232255 23495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } }
I20251025 20:08:25.232405 23495 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:25.232472 23495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:25.232545 23495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:25.233210 23495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } }
I20251025 20:08:25.233491 23495 leader_election.cc:304] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bab2cf528ee48f6a9febcdb5d79b16b; no voters:
I20251025 20:08:25.233745 23495 leader_election.cc:290] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:25.234040 23500 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:25.234218 23500 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [term 1 LEADER]: Becoming Leader. State: Replica: 5bab2cf528ee48f6a9febcdb5d79b16b, State: Running, Role: LEADER
I20251025 20:08:25.234496 23500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } }
I20251025 20:08:25.234678 23495 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:25.235805 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bab2cf528ee48f6a9febcdb5d79b16b. Latest consensus state: current_term: 1 leader_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } } }
I20251025 20:08:25.235917 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:25.236164 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bab2cf528ee48f6a9febcdb5d79b16b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 37943 } } }
I20251025 20:08:25.236214 23509 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:25.236243 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:25.238795 23509 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:25.243765 23509 catalog_manager.cc:1357] Generated new cluster ID: 0040c7508ab54dd086f229e7b1863ca7
I20251025 20:08:25.243849 23509 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:25.257472 23509 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:25.258548 23509 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:25.265902 23509 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: Generated new TSK 0
I20251025 20:08:25.266654 23509 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:25.310860 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:37943
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:25.411837 23523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:25.412029 23523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:25.412115 23523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:25.414860 23523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:25.414911 23523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:25.415009 23523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:25.418761 23523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:37943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:25.419723 23523 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:25.420537 23523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:25.426566 23529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.426636 23523 server_base.cc:1047] running on GCE node
W20251025 20:08:25.426540 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.426555 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.427064 23523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:25.427577 23523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:25.428736 23523 hybrid_clock.cc:648] HybridClock initialized: now 1761422905428718 us; error 29 us; skew 500 ppm
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:25.431149 23523 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:25.431864 23523 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:25.431928 23523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:25.432113 23523 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:25.433774 23523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "37970511687b427baa0b8fedde855546"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "91d76544d81c518e541ec96aaa8a9403"
server_key_iv: "a52e430341d28786e0eaeb6b7f1836a5"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.434218 23523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "37970511687b427baa0b8fedde855546"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "91d76544d81c518e541ec96aaa8a9403"
server_key_iv: "a52e430341d28786e0eaeb6b7f1836a5"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.437227 23523 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:08:25.439230 23537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.440115 23523 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:25.440220 23523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "37970511687b427baa0b8fedde855546"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "91d76544d81c518e541ec96aaa8a9403"
server_key_iv: "a52e430341d28786e0eaeb6b7f1836a5"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.440302 23523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:25.453313 23523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:25.455806 23523 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:25.455955 23523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:25.456422 23523 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:25.457178 23523 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:25.457222 23523 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.457275 23523 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:25.457306 23523 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.465934 23523 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:36963
I20251025 20:08:25.465965 23650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:36963 every 8 connection(s)
I20251025 20:08:25.466462 23523 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:25.476058 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23523
I20251025 20:08:25.476162 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:25.476392 18761 external_mini_cluster.cc:1468] Setting key bbfd4f6ef2367ba47e34e34080a0be29
I20251025 20:08:25.477607 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.467980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:36857 (local address 127.18.82.126:37943)
1025 20:08:25.468094 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.468097 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.468673 (+ 576us) server_negotiation.cc:408] Connection header received
1025 20:08:25.469446 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.469450 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.469535 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.469626 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:25.471207 (+ 1581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.471739 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.472331 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.472489 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.474551 (+ 2062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:25.474568 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:25.474571 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:25.474606 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:25.475767 (+ 1161us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.476208 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.476211 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.476212 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.476267 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.476553 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.476557 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.476559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.476738 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:25.476834 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.477323 (+ 489us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.477439 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251025 20:08:25.478199 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.468170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:37943 (local address 127.18.82.65:36857)
1025 20:08:25.468542 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.468574 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.469245 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.469769 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.469777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.470154 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:25.471053 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.471067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.471847 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.471849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.472217 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.472224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.472414 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.473025 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:25.473046 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:25.474410 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:25.475891 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.475896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.475903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.476100 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.476349 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.476353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.476363 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.476442 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.476852 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:25.476858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:25.477077 (+ 219us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.477274 (+ 197us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.477527 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20251025 20:08:25.479343 23651 heartbeater.cc:344] Connected to a master server at 127.18.82.126:37943
I20251025 20:08:25.479576 23651 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:25.480001 23651 heartbeater.cc:507] Master 127.18.82.126:37943 requested a full tablet report, sending...
I20251025 20:08:25.481395 23459 ts_manager.cc:194] Registered new tserver with Master: 37970511687b427baa0b8fedde855546 (127.18.82.65:36963)
I20251025 20:08:25.482519 23459 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:36857
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:25.518611 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:37943
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:25.618160 23658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:25.618340 23658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:25.618459 23658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:25.621485 23658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:25.621568 23658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:25.621670 23658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:25.625373 23658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:37943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:25.626277 23658 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:25.627100 23658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:25.632808 23663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.632973 23658 server_base.cc:1047] running on GCE node
W20251025 20:08:25.633014 23664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.632826 23666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.633488 23658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:25.634047 23658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:25.635206 23658 hybrid_clock.cc:648] HybridClock initialized: now 1761422905635137 us; error 78 us; skew 500 ppm
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:25.637989 23658 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:25.638629 23658 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:25.638689 23658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:25.638886 23658 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:25.640470 23658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "ebef4f9dcb404c8db5e27d33dd912a58"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "177dae0dce2d7801df5f90286e2f8538"
server_key_iv: "221ee38221a548c60aee6c9a344fda5d"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.640931 23658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "ebef4f9dcb404c8db5e27d33dd912a58"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "177dae0dce2d7801df5f90286e2f8538"
server_key_iv: "221ee38221a548c60aee6c9a344fda5d"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.644083 23658 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20251025 20:08:25.646154 23672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.647095 23658 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:25.647200 23658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "ebef4f9dcb404c8db5e27d33dd912a58"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "177dae0dce2d7801df5f90286e2f8538"
server_key_iv: "221ee38221a548c60aee6c9a344fda5d"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.647285 23658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:25.675493 23658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:25.677778 23658 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:25.677933 23658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:25.678406 23658 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:25.679180 23658 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:25.679226 23658 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.679278 23658 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:25.679308 23658 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.687659 23658 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:39257
I20251025 20:08:25.687680 23785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:39257 every 8 connection(s)
I20251025 20:08:25.688131 23658 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:25.692942 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23658
I20251025 20:08:25.693060 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:25.693272 18761 external_mini_cluster.cc:1468] Setting key 3d578427e407522bf575ba024405af12
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:25.700043 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.689605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:46245 (local address 127.18.82.126:37943)
1025 20:08:25.689743 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.689747 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.690404 (+ 657us) server_negotiation.cc:408] Connection header received
1025 20:08:25.691225 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.691229 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.691305 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.691377 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:25.692878 (+ 1501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.693432 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.694230 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.694387 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.696460 (+ 2073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:25.696482 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:25.696487 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:25.696520 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:25.698146 (+ 1626us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.698658 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.698664 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.698667 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.698724 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.698983 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.698989 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.698993 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.699184 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:25.699294 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.699693 (+ 399us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.699827 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 20:08:25.700493 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.689841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:37943 (local address 127.18.82.66:46245)
1025 20:08:25.690262 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.690297 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.691024 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.691547 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.691554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.691876 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:25.692724 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.692737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.693633 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.693638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.694099 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.694110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.694308 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.694894 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:25.694927 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:25.696263 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:25.698251 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.698255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.698263 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.698551 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.698803 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.698805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.698806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.698902 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.699276 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:25.699280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:25.699473 (+ 193us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.699624 (+ 151us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.699807 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20251025 20:08:25.701537 23786 heartbeater.cc:344] Connected to a master server at 127.18.82.126:37943
I20251025 20:08:25.701782 23786 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:25.702322 23786 heartbeater.cc:507] Master 127.18.82.126:37943 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:08:25.703364 23459 ts_manager.cc:194] Registered new tserver with Master: ebef4f9dcb404c8db5e27d33dd912a58 (127.18.82.66:39257)
I20251025 20:08:25.703939 23459 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:46245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:25.736449 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:37943
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:25.833739 23793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:25.833922 23793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:25.833995 23793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:25.837097 23793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:25.837147 23793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:25.837250 23793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:25.840962 23793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:37943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:25.841915 23793 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:25.842676 23793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:25.848565 23799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.848627 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:25.848562 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:25.848752 23793 server_base.cc:1047] running on GCE node
I20251025 20:08:25.849170 23793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:25.849704 23793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:25.850847 23793 hybrid_clock.cc:648] HybridClock initialized: now 1761422905850816 us; error 44 us; skew 500 ppm
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:25.853278 23793 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:25.853946 23793 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:25.854012 23793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:25.854182 23793 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:25.855831 23793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "97c9bca8f4564159bcef3b9dd8321d1e"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "20a56863823531dc53fd7e22cceec8d0"
server_key_iv: "01d43c4148b9c2886b672f478e71f5d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.856276 23793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "97c9bca8f4564159bcef3b9dd8321d1e"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "20a56863823531dc53fd7e22cceec8d0"
server_key_iv: "01d43c4148b9c2886b672f478e71f5d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.859349 23793 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:25.861341 23807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.862275 23793 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:25.862373 23793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "97c9bca8f4564159bcef3b9dd8321d1e"
format_stamp: "Formatted at 2025-10-25 20:08:25 on dist-test-slave-d2pk"
server_key: "20a56863823531dc53fd7e22cceec8d0"
server_key_iv: "01d43c4148b9c2886b672f478e71f5d4"
server_key_version: "encryptionkey@0"
I20251025 20:08:25.862453 23793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:25.882750 23793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:25.885387 23793 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:25.885583 23793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:25.886092 23793 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:25.886952 23793 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:25.887001 23793 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.887060 23793 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:25.887094 23793 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:25.895332 23793 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:44583
I20251025 20:08:25.895378 23920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:44583 every 8 connection(s)
I20251025 20:08:25.895833 23793 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:25.901036 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23793
I20251025 20:08:25.901165 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:25.901418 18761 external_mini_cluster.cc:1468] Setting key 0a8f4249a81f1bf679d75408e6c4e2fa
Oct 25 20:08:25 dist-test-slave-d2pk krb5kdc[23412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:25.906589 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.897278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:32975 (local address 127.18.82.126:37943)
1025 20:08:25.897402 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.897405 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.898044 (+ 639us) server_negotiation.cc:408] Connection header received
1025 20:08:25.898800 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.898804 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.898852 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.898919 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:25.900403 (+ 1484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.900945 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.901681 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.901824 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.903945 (+ 2121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:25.903959 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:25.903961 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:25.903985 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:25.905120 (+ 1135us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.905541 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.905546 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.905548 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.905585 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.905798 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.905800 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.905801 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.905923 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:25.905990 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.906364 (+ 374us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.906480 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:08:25.907227 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.897482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:37943 (local address 127.18.82.67:32975)
1025 20:08:25.897891 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.897921 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.898633 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.899039 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.899046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.899400 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:25.900260 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.900274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.901050 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.901052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.901565 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.901573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.901719 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.902286 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:25.902318 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:25.903817 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:25.905215 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.905219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.905227 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.905440 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.905654 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:25.905656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:25.905658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:25.905733 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:25.905998 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:25.906002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:25.906182 (+ 180us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.906343 (+ 161us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.906565 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
I20251025 20:08:25.908141 23921 heartbeater.cc:344] Connected to a master server at 127.18.82.126:37943
I20251025 20:08:25.908340 23921 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:25.908737 23921 heartbeater.cc:507] Master 127.18.82.126:37943 requested a full tablet report, sending...
I20251025 20:08:25.909579 23459 ts_manager.cc:194] Registered new tserver with Master: 97c9bca8f4564159bcef3b9dd8321d1e (127.18.82.67:44583)
I20251025 20:08:25.910063 23459 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:32975
I20251025 20:08:25.914789 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:08:25.921451 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.916329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.18.82.126:37943)
1025 20:08:25.916440 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.916443 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.916564 (+ 121us) server_negotiation.cc:408] Connection header received
1025 20:08:25.916651 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.916653 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.916685 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.916744 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:25.917446 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.917886 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.918387 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.918499 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.919343 (+ 844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:25.919355 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:25.919356 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:25.919373 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:25.920405 (+ 1032us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.920727 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.920729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.920730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.920761 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:25.920905 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:25.920907 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:25.920908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:25.921038 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:25.921132 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.921263 (+ 131us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.921352 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:08:25.923645 23459 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:25.925992 23459 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:25.937840 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.933758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:44583 (local address 127.0.0.1:59278)
1025 20:08:25.934482 (+ 724us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.934494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.934562 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.935205 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.935208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.935226 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:25.935398 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.935402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.936508 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.936512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.937335 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.937343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.937452 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.937499 (+ 47us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.937603 (+ 104us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.937683 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":77,"threads_started":1}
I20251025 20:08:25.937840 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.933503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36963 (local address 127.0.0.1:47840)
1025 20:08:25.934105 (+ 602us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.934137 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.934260 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.935093 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.935099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.935134 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:25.935402 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.935410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.936595 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.936598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.937334 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.937343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.937452 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.937499 (+ 47us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.937603 (+ 104us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.937683 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"spinlock_wait_cycles":5248,"thread_start_us":70,"threads_started":1}
I20251025 20:08:25.938380 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.933702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.18.82.65:36963)
1025 20:08:25.934635 (+ 933us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.934642 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.934665 (+ 23us) server_negotiation.cc:408] Connection header received
1025 20:08:25.934748 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.934753 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.934943 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.935071 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:25.935550 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.936498 (+ 948us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.937479 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.937983 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.938086 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.938145 (+ 59us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.938221 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":110,"threads_started":1}
I20251025 20:08:25.939244 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.934131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:39257 (local address 127.0.0.1:46284)
1025 20:08:25.935041 (+ 910us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:25.935054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:25.935139 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:25.935582 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:25.935585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:25.935601 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:25.935789 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.935795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.937061 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.937064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:25.937847 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:25.937858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.938960 (+ 1102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.938976 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 20:08:25.939059 (+ 83us) client_negotiation.cc:240] Negotiation successful
1025 20:08:25.939117 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"spinlock_wait_cycles":4480,"thread_start_us":76,"threads_started":1}
I20251025 20:08:25.939438 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.934191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.18.82.67:44583)
1025 20:08:25.934863 (+ 672us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.934868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.934884 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:08:25.934945 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.934949 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.935075 (+ 126us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.935519 (+ 444us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:25.935658 (+ 139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.936370 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.938570 (+ 2200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.939059 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.939158 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.939213 (+ 55us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.939291 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":130,"threads_started":1}
I20251025 20:08:25.940486 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:25.934530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.18.82.66:39257)
1025 20:08:25.934929 (+ 399us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:25.934934 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:25.935224 (+ 290us) server_negotiation.cc:408] Connection header received
1025 20:08:25.935304 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:25.935309 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:25.935460 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:25.935576 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:25.935889 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.936945 (+ 1056us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:25.938281 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:25.938801 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:25.938894 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:25.940244 (+ 1350us) server_negotiation.cc:300] Negotiation successful
1025 20:08:25.940339 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":119,"threads_started":1}
I20251025 20:08:25.941154 23585 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca4acabab8e45238867623ca017dfaa (DEFAULT_TABLE table=test-table [id=4cf871650f7746108eada216c9fc8d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:25.942158 23585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca4acabab8e45238867623ca017dfaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:25.941975 23855 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca4acabab8e45238867623ca017dfaa (DEFAULT_TABLE table=test-table [id=4cf871650f7746108eada216c9fc8d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:25.942466 23720 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca4acabab8e45238867623ca017dfaa (DEFAULT_TABLE table=test-table [id=4cf871650f7746108eada216c9fc8d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:25.942770 23855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca4acabab8e45238867623ca017dfaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:25.943105 23720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca4acabab8e45238867623ca017dfaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:25.947712 23938 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Bootstrap starting.
I20251025 20:08:25.948657 23940 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Bootstrap starting.
I20251025 20:08:25.948683 23939 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Bootstrap starting.
I20251025 20:08:25.950227 23938 tablet_bootstrap.cc:654] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:25.950529 23940 tablet_bootstrap.cc:654] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:25.950528 23939 tablet_bootstrap.cc:654] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:25.950923 23938 log.cc:826] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:25.951304 23939 log.cc:826] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:25.951304 23940 log.cc:826] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:25.952620 23938 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: No bootstrap required, opened a new log
I20251025 20:08:25.952802 23938 ts_tablet_manager.cc:1403] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:08:25.952872 23940 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: No bootstrap required, opened a new log
I20251025 20:08:25.953029 23940 ts_tablet_manager.cc:1403] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:08:25.954485 23939 tablet_bootstrap.cc:492] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: No bootstrap required, opened a new log
I20251025 20:08:25.954649 23939 ts_tablet_manager.cc:1403] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251025 20:08:25.956290 23938 raft_consensus.cc:359] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.956290 23940 raft_consensus.cc:359] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.956486 23938 raft_consensus.cc:385] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:25.956486 23940 raft_consensus.cc:385] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:25.956535 23940 raft_consensus.cc:740] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c9bca8f4564159bcef3b9dd8321d1e, State: Initialized, Role: FOLLOWER
I20251025 20:08:25.956535 23938 raft_consensus.cc:740] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebef4f9dcb404c8db5e27d33dd912a58, State: Initialized, Role: FOLLOWER
I20251025 20:08:25.956972 23940 consensus_queue.cc:260] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.956972 23938 consensus_queue.cc:260] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.957129 23939 raft_consensus.cc:359] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.957301 23939 raft_consensus.cc:385] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:25.957371 23939 raft_consensus.cc:740] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37970511687b427baa0b8fedde855546, State: Initialized, Role: FOLLOWER
I20251025 20:08:25.957789 23940 ts_tablet_manager.cc:1434] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 20:08:25.957789 23938 ts_tablet_manager.cc:1434] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20251025 20:08:25.957806 23939 consensus_queue.cc:260] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:25.957919 23921 heartbeater.cc:499] Master 127.18.82.126:37943 was elected leader, sending a full tablet report...
I20251025 20:08:25.958086 23786 heartbeater.cc:499] Master 127.18.82.126:37943 was elected leader, sending a full tablet report...
I20251025 20:08:25.958659 23939 ts_tablet_manager.cc:1434] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:08:25.959003 23651 heartbeater.cc:499] Master 127.18.82.126:37943 was elected leader, sending a full tablet report...
W20251025 20:08:25.967311 23652 tablet.cc:2378] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:26.001672 23944 raft_consensus.cc:493] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:26.001832 23944 raft_consensus.cc:515] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:26.002894 23944 leader_election.cc:290] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37970511687b427baa0b8fedde855546 (127.18.82.65:36963), 97c9bca8f4564159bcef3b9dd8321d1e (127.18.82.67:44583)
I20251025 20:08:26.005596 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.003084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:44583 (local address 127.18.82.66:42821)
1025 20:08:26.003192 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.003208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.003295 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.003600 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.003603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.003628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:26.003878 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:26.003886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.004668 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:26.004671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:26.005268 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:26.005276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.005370 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:26.005388 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.005422 (+ 34us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.005462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20251025 20:08:26.005860 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.003084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36963 (local address 127.18.82.66:55289)
1025 20:08:26.003505 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.003520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.003626 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.003876 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.003878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.003893 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:26.004183 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:26.004191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.004897 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:26.004900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:26.005460 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:26.005469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.005603 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:26.005628 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.005677 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.005730 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":103,"thread_start_us":140,"threads_started":1}
I20251025 20:08:26.006138 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.003159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:42821 (local address 127.18.82.67:44583)
1025 20:08:26.003276 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.003281 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.003296 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:26.003429 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.003433 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.003483 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.003574 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:26.004000 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.004565 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:26.005397 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.005864 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:26.005906 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.005962 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.006017 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:26.006249 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.003159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:55289 (local address 127.18.82.65:36963)
1025 20:08:26.003276 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.003281 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.003600 (+ 319us) server_negotiation.cc:408] Connection header received
1025 20:08:26.003735 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.003740 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.003790 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.003861 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:26.004287 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.004786 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:26.005594 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.006048 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:26.006080 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.006120 (+ 40us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.006160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:26.006781 23875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca4acabab8e45238867623ca017dfaa" candidate_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" is_pre_election: true
I20251025 20:08:26.006851 23605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca4acabab8e45238867623ca017dfaa" candidate_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37970511687b427baa0b8fedde855546" is_pre_election: true
I20251025 20:08:26.007057 23875 raft_consensus.cc:2468] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebef4f9dcb404c8db5e27d33dd912a58 in term 0.
I20251025 20:08:26.007103 23605 raft_consensus.cc:2468] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebef4f9dcb404c8db5e27d33dd912a58 in term 0.
I20251025 20:08:26.007508 23676 leader_election.cc:304] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97c9bca8f4564159bcef3b9dd8321d1e, ebef4f9dcb404c8db5e27d33dd912a58; no voters:
I20251025 20:08:26.007727 23944 raft_consensus.cc:2804] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:26.007792 23944 raft_consensus.cc:493] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:26.007836 23944 raft_consensus.cc:3060] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.008692 23944 raft_consensus.cc:515] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:26.008955 23944 leader_election.cc:290] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [CANDIDATE]: Term 1 election: Requested vote from peers 37970511687b427baa0b8fedde855546 (127.18.82.65:36963), 97c9bca8f4564159bcef3b9dd8321d1e (127.18.82.67:44583)
I20251025 20:08:26.009268 23605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca4acabab8e45238867623ca017dfaa" candidate_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37970511687b427baa0b8fedde855546"
I20251025 20:08:26.009310 23875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca4acabab8e45238867623ca017dfaa" candidate_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c9bca8f4564159bcef3b9dd8321d1e"
I20251025 20:08:26.009364 23605 raft_consensus.cc:3060] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.009465 23875 raft_consensus.cc:3060] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.010305 23605 raft_consensus.cc:2468] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebef4f9dcb404c8db5e27d33dd912a58 in term 1.
I20251025 20:08:26.010414 23875 raft_consensus.cc:2468] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebef4f9dcb404c8db5e27d33dd912a58 in term 1.
I20251025 20:08:26.010653 23675 leader_election.cc:304] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37970511687b427baa0b8fedde855546, ebef4f9dcb404c8db5e27d33dd912a58; no voters:
I20251025 20:08:26.010859 23944 raft_consensus.cc:2804] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:26.011050 23944 raft_consensus.cc:697] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 1 LEADER]: Becoming Leader. State: Replica: ebef4f9dcb404c8db5e27d33dd912a58, State: Running, Role: LEADER
I20251025 20:08:26.011371 23944 consensus_queue.cc:237] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } }
I20251025 20:08:26.014190 23458 catalog_manager.cc:5649] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebef4f9dcb404c8db5e27d33dd912a58 (127.18.82.66). New cstate: current_term: 1 leader_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebef4f9dcb404c8db5e27d33dd912a58" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 39257 } health_report { overall_health: HEALTHY } } }
I20251025 20:08:26.033183 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.030453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.18.82.66:39257)
1025 20:08:26.030572 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.030576 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.030588 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:26.030663 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.030666 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.030716 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.030793 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:26.031064 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.031545 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:26.032110 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:26.032288 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:26.032416 (+ 128us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:26.032826 (+ 410us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:26.032933 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.033011 (+ 78us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.033059 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:26.038352 23605 raft_consensus.cc:1275] T 0ca4acabab8e45238867623ca017dfaa P 37970511687b427baa0b8fedde855546 [term 1 FOLLOWER]: Refusing update from remote peer ebef4f9dcb404c8db5e27d33dd912a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:26.038352 23875 raft_consensus.cc:1275] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 1 FOLLOWER]: Refusing update from remote peer ebef4f9dcb404c8db5e27d33dd912a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:26.038940 23944 consensus_queue.cc:1048] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c9bca8f4564159bcef3b9dd8321d1e" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:26.039098 23948 consensus_queue.cc:1048] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37970511687b427baa0b8fedde855546" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:26.053083 23456 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45926:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:26.053206 23456 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45926:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:26.055220 23456 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5bab2cf528ee48f6a9febcdb5d79b16b: Sending DeleteTablet for 3 replicas of tablet 0ca4acabab8e45238867623ca017dfaa
I20251025 20:08:26.055884 23855 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ca4acabab8e45238867623ca017dfaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:59278
I20251025 20:08:26.055877 23585 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ca4acabab8e45238867623ca017dfaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:47840
I20251025 20:08:26.056432 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23523
I20251025 20:08:26.056476 23720 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ca4acabab8e45238867623ca017dfaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:46284
I20251025 20:08:26.057207 23960 tablet_replica.cc:333] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: stopping tablet replica
I20251025 20:08:26.057546 23960 raft_consensus.cc:2243] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:26.057893 23962 tablet_replica.cc:333] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: stopping tablet replica
I20251025 20:08:26.057889 23960 raft_consensus.cc:2272] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:26.058149 23962 raft_consensus.cc:2243] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:26.058766 23962 raft_consensus.cc:2272] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:26.059111 23960 ts_tablet_manager.cc:1916] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:26.061038 23960 ts_tablet_manager.cc:1929] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:26.061121 23960 log.cc:1199] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/0ca4acabab8e45238867623ca017dfaa
I20251025 20:08:26.061502 23960 ts_tablet_manager.cc:1950] T 0ca4acabab8e45238867623ca017dfaa P 97c9bca8f4564159bcef3b9dd8321d1e: Deleting consensus metadata
I20251025 20:08:26.062878 23446 catalog_manager.cc:4985] TS 97c9bca8f4564159bcef3b9dd8321d1e (127.18.82.67:44583): tablet 0ca4acabab8e45238867623ca017dfaa (table test-table [id=4cf871650f7746108eada216c9fc8d84]) successfully deleted
I20251025 20:08:26.063555 23962 ts_tablet_manager.cc:1916] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:26.065369 23962 ts_tablet_manager.cc:1929] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:26.065555 23962 log.cc:1199] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/0ca4acabab8e45238867623ca017dfaa
W20251025 20:08:26.065670 23445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:36963 (error 108)
I20251025 20:08:26.065953 23962 ts_tablet_manager.cc:1950] T 0ca4acabab8e45238867623ca017dfaa P ebef4f9dcb404c8db5e27d33dd912a58: Deleting consensus metadata
I20251025 20:08:26.066485 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23658
I20251025 20:08:26.070327 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.069962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36963 (local address 127.0.0.1:47844)
1025 20:08:26.070132 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.070226 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251025 20:08:26.070487 23445 catalog_manager.cc:4712] TS 37970511687b427baa0b8fedde855546 (127.18.82.65:36963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ca4acabab8e45238867623ca017dfaa: Network error: Client connection negotiation failed: client connection to 127.18.82.65:36963: connect: Connection refused (error 111)
I20251025 20:08:26.072639 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23793
I20251025 20:08:26.073589 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.072748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:39257 (local address 127.0.0.1:46310)
1025 20:08:26.073452 (+ 704us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.073491 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.66:39257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635}
I20251025 20:08:26.078473 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23428
2025-10-25T20:08:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3002 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-25T20:08:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:26Z Disabled control of system clock
I20251025 20:08:26.107265 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43743
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251025 20:08:26.203881 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:26.204044 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:26.204084 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:26.206750 23967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:26.206799 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:26.206815 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:26.206832 23967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:26.206854 23967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:26.210011 23967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:43743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:26.210971 23967 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:26.212025 23967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:26.217060 23975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.217064 23972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.217067 23973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.217240 23967 server_base.cc:1047] running on GCE node
I20251025 20:08:26.217756 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:26.218617 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:26.219765 23967 hybrid_clock.cc:648] HybridClock initialized: now 1761422906219752 us; error 34 us; skew 500 ppm
I20251025 20:08:26.221421 23967 webserver.cc:492] Webserver started at http://127.18.82.126:45369/ using document root <none> and password file <none>
I20251025 20:08:26.221915 23967 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:26.221972 23967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:26.222129 23967 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:26.223625 23967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "a3567920a9224d8ab37340a7ce56d0a4"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "c0d9b8016e48e197ecfb8e9467d383e9"
server_key_iv: "57bd4263311b42c5af500604c37902e1"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.224076 23967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "a3567920a9224d8ab37340a7ce56d0a4"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "c0d9b8016e48e197ecfb8e9467d383e9"
server_key_iv: "57bd4263311b42c5af500604c37902e1"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.227074 23967 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:26.228943 23981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.229914 23967 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:26.230018 23967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "a3567920a9224d8ab37340a7ce56d0a4"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "c0d9b8016e48e197ecfb8e9467d383e9"
server_key_iv: "57bd4263311b42c5af500604c37902e1"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.230101 23967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:26.234484 23967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:26.234964 23967 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:26.235100 23967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:26.241355 24033 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:43743 every 8 connection(s)
I20251025 20:08:26.241353 23967 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:43743
I20251025 20:08:26.242251 23967 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:26.244725 24034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:26.249541 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: Bootstrap starting.
I20251025 20:08:26.251430 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 23967
I20251025 20:08:26.251559 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:26.251662 24034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:26.251796 18761 external_mini_cluster.cc:1468] Setting key eaf3922b4462cbbdc6d1a4be4df9a9c3
I20251025 20:08:26.252336 24034 log.cc:826] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:26.254261 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: No bootstrap required, opened a new log
I20251025 20:08:26.256294 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.252701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.18.82.126:43743)
1025 20:08:26.253259 (+ 558us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.253269 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.253305 (+ 36us) server_negotiation.cc:408] Connection header received
1025 20:08:26.253921 (+ 616us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.253944 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.254221 (+ 277us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.254542 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.255172 (+ 630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.255177 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.255187 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.255197 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.255239 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.255251 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.255348 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.255454 (+ 106us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.255652 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20251025 20:08:26.257153 24034 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } }
I20251025 20:08:26.257339 24034 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:26.257403 24034 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3567920a9224d8ab37340a7ce56d0a4, State: Initialized, Role: FOLLOWER
I20251025 20:08:26.257824 24034 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } }
I20251025 20:08:26.257933 24034 raft_consensus.cc:399] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:26.257989 24034 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:26.258077 24034 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.258816 24034 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } }
I20251025 20:08:26.259101 24034 leader_election.cc:304] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3567920a9224d8ab37340a7ce56d0a4; no voters:
I20251025 20:08:26.259336 24034 leader_election.cc:290] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:26.259433 24039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:26.259591 24039 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [term 1 LEADER]: Becoming Leader. State: Replica: a3567920a9224d8ab37340a7ce56d0a4, State: Running, Role: LEADER
I20251025 20:08:26.259852 24039 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } }
I20251025 20:08:26.260437 24034 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:26.261422 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3567920a9224d8ab37340a7ce56d0a4. Latest consensus state: current_term: 1 leader_uuid: "a3567920a9224d8ab37340a7ce56d0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } } }
I20251025 20:08:26.261564 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:26.261813 24040 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3567920a9224d8ab37340a7ce56d0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3567920a9224d8ab37340a7ce56d0a4" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43743 } } }
I20251025 20:08:26.261900 24040 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:26.262205 24048 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:26.264672 24048 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:26.269465 24048 catalog_manager.cc:1357] Generated new cluster ID: a4199ed733404741b850b56ad3a7d130
I20251025 20:08:26.269539 24048 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:26.280951 24048 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:26.281230 24048 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:26.309711 24048 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: Generated new TSK 0
I20251025 20:08:26.310436 24048 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:08:26.324331 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--builtin_ntp_servers=127.18.82.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251025 20:08:26.424438 24058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:26.424605 24058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:26.424650 24058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:26.427624 24058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:26.427662 24058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:26.427735 24058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:26.430963 24058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:26.431891 24058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:26.432875 24058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:26.438480 24064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.438606 24058 server_base.cc:1047] running on GCE node
W20251025 20:08:26.438541 24066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.438786 24063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.439071 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:26.439528 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:26.440668 24058 hybrid_clock.cc:648] HybridClock initialized: now 1761422906440650 us; error 35 us; skew 500 ppm
I20251025 20:08:26.442217 24058 webserver.cc:492] Webserver started at http://127.18.82.65:35967/ using document root <none> and password file <none>
I20251025 20:08:26.442750 24058 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:26.442813 24058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:26.442982 24058 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:26.444466 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "380bffbec91f4369af502c0c9f4b2073"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "98757e7acf02f89cc1b34de3ac58b092"
server_key_iv: "23a4fe39d6e6d684a8d68c8a0feb1e69"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.444900 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "380bffbec91f4369af502c0c9f4b2073"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "98757e7acf02f89cc1b34de3ac58b092"
server_key_iv: "23a4fe39d6e6d684a8d68c8a0feb1e69"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.447824 24058 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:26.449896 24072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.450783 24058 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:26.450886 24058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "380bffbec91f4369af502c0c9f4b2073"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "98757e7acf02f89cc1b34de3ac58b092"
server_key_iv: "23a4fe39d6e6d684a8d68c8a0feb1e69"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.450971 24058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:26.454996 24058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:26.455473 24058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:26.455608 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:26.456043 24058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:26.456897 24058 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:26.456943 24058 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.456991 24058 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:26.457021 24058 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.465296 24058 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:33267
I20251025 20:08:26.465353 24185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:33267 every 8 connection(s)
I20251025 20:08:26.466167 24058 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:26.469375 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24058
I20251025 20:08:26.469483 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:26.469755 18761 external_mini_cluster.cc:1468] Setting key b25f5450e528d2b6eb9967c986729ab8
I20251025 20:08:26.471433 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.467884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:43137 (local address 127.18.82.126:43743)
1025 20:08:26.468028 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.468031 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.468897 (+ 866us) server_negotiation.cc:408] Connection header received
1025 20:08:26.469711 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.469715 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.469767 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.469853 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.470935 (+ 1082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.470939 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.470942 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.470955 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.470980 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.470987 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.471066 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.471209 (+ 143us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.471254 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:26.471685 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--builtin_ntp_servers=127.18.82.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251025 20:08:26.472079 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.468105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43743 (local address 127.18.82.65:43137)
1025 20:08:26.468765 (+ 660us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.468796 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.469456 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.470017 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.470024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.470102 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.470710 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.470712 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.470744 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.470793 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.471063 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.471071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.471078 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.471268 (+ 190us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.471492 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20251025 20:08:26.473327 24186 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43743
I20251025 20:08:26.473594 24186 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:26.473982 24186 heartbeater.cc:507] Master 127.18.82.126:43743 requested a full tablet report, sending...
I20251025 20:08:26.475309 23998 ts_manager.cc:194] Registered new tserver with Master: 380bffbec91f4369af502c0c9f4b2073 (127.18.82.65:33267)
W20251025 20:08:26.568936 24189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:26.569118 24189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:26.569188 24189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:26.572252 24189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:26.572294 24189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:26.572394 24189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:26.575704 24189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:26.576612 24189 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:26.577693 24189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:26.583817 24197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.583811 24194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.584067 24189 server_base.cc:1047] running on GCE node
W20251025 20:08:26.584113 24195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.584430 24189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:26.584904 24189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:26.586052 24189 hybrid_clock.cc:648] HybridClock initialized: now 1761422906586035 us; error 33 us; skew 500 ppm
I20251025 20:08:26.587780 24189 webserver.cc:492] Webserver started at http://127.18.82.66:45789/ using document root <none> and password file <none>
I20251025 20:08:26.588270 24189 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:26.588327 24189 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:26.588493 24189 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:26.590102 24189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "f481d9a4099949538764fe0700799f79"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "87a90f4a5af1e986f12fc077572f4988"
server_key_iv: "24c345f80ada332b2ea410a13e9a1522"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.590579 24189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "f481d9a4099949538764fe0700799f79"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "87a90f4a5af1e986f12fc077572f4988"
server_key_iv: "24c345f80ada332b2ea410a13e9a1522"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.593716 24189 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:08:26.595777 24203 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.596654 24189 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:26.596761 24189 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "f481d9a4099949538764fe0700799f79"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "87a90f4a5af1e986f12fc077572f4988"
server_key_iv: "24c345f80ada332b2ea410a13e9a1522"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.596843 24189 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:26.601111 24189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:26.601670 24189 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:26.601807 24189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:26.602268 24189 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:26.603075 24189 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:26.603117 24189 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.603158 24189 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:26.603188 24189 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.611084 24189 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:37115
I20251025 20:08:26.611121 24316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:37115 every 8 connection(s)
I20251025 20:08:26.611994 24189 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:26.616511 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.613451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:42351 (local address 127.18.82.126:43743)
1025 20:08:26.613603 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.613607 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.614341 (+ 734us) server_negotiation.cc:408] Connection header received
1025 20:08:26.615051 (+ 710us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.615054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.615093 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.615164 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.616185 (+ 1021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.616188 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.616190 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.616200 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.616219 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.616225 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.616297 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.616383 (+ 86us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.616414 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:26.617091 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24189
I20251025 20:08:26.617123 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.613712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43743 (local address 127.18.82.66:42351)
1025 20:08:26.614227 (+ 515us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.614254 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.614890 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.615306 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.615313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.615389 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.615990 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.615992 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.616017 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.616064 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.616288 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.616291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.616294 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.616424 (+ 130us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.616582 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20251025 20:08:26.617188 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:26.617400 18761 external_mini_cluster.cc:1468] Setting key ad83256070dbc3acdb05ea5d7d0563a2
I20251025 20:08:26.618472 24317 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43743
I20251025 20:08:26.618692 24317 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:26.619081 24317 heartbeater.cc:507] Master 127.18.82.126:43743 requested a full tablet report, sending...
I20251025 20:08:26.619273 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--builtin_ntp_servers=127.18.82.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251025 20:08:26.620062 23998 ts_manager.cc:194] Registered new tserver with Master: f481d9a4099949538764fe0700799f79 (127.18.82.66:37115)
W20251025 20:08:26.719659 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:26.719849 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:26.719931 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:26.722894 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:26.722942 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:26.723039 24320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:26.726550 24320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:26.727557 24320 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:26.728605 24320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:26.734769 24328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.734776 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:26.734753 24325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:26.734954 24320 server_base.cc:1047] running on GCE node
I20251025 20:08:26.735287 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:26.735787 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:26.736929 24320 hybrid_clock.cc:648] HybridClock initialized: now 1761422906736923 us; error 26 us; skew 500 ppm
I20251025 20:08:26.738483 24320 webserver.cc:492] Webserver started at http://127.18.82.67:40355/ using document root <none> and password file <none>
I20251025 20:08:26.738982 24320 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:26.739041 24320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:26.739204 24320 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:26.740774 24320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "e375d595440549cf985f5fd340f8fe5b"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "1854a82d9961f7b673ac99689a524329"
server_key_iv: "be53c5e1525dd5d4f4b6f79f326db5ca"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.741235 24320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "e375d595440549cf985f5fd340f8fe5b"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "1854a82d9961f7b673ac99689a524329"
server_key_iv: "be53c5e1525dd5d4f4b6f79f326db5ca"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.744372 24320 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:26.746330 24334 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.747282 24320 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:26.747395 24320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "e375d595440549cf985f5fd340f8fe5b"
format_stamp: "Formatted at 2025-10-25 20:08:26 on dist-test-slave-d2pk"
server_key: "1854a82d9961f7b673ac99689a524329"
server_key_iv: "be53c5e1525dd5d4f4b6f79f326db5ca"
server_key_version: "encryptionkey@0"
I20251025 20:08:26.747475 24320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:26.751814 24320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:26.752295 24320 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:26.752439 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:26.752895 24320 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:26.753829 24320 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:26.753880 24320 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.753934 24320 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:26.753962 24320 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:26.762488 24320 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:37629
I20251025 20:08:26.762534 24447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:37629 every 8 connection(s)
I20251025 20:08:26.763326 24320 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:26.764515 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24320
I20251025 20:08:26.764629 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:26.764891 18761 external_mini_cluster.cc:1468] Setting key 327e8207b34bdd9c5986b342b0786903
I20251025 20:08:26.768641 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.764920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:54941 (local address 127.18.82.126:43743)
1025 20:08:26.765299 (+ 379us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.765304 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.766134 (+ 830us) server_negotiation.cc:408] Connection header received
1025 20:08:26.767220 (+ 1086us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.767223 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.767261 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.767326 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.768274 (+ 948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.768277 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.768280 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.768291 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.768311 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.768318 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.768386 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.768499 (+ 113us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.768532 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20251025 20:08:26.769323 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.765300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43743 (local address 127.18.82.67:54941)
1025 20:08:26.766017 (+ 717us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.766056 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.766977 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.767445 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.767452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.767530 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.768071 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.768073 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.768102 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.768153 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.768392 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.768396 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.768400 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.768546 (+ 146us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.768728 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
I20251025 20:08:26.770491 24448 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43743
I20251025 20:08:26.770702 24448 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:26.771063 24448 heartbeater.cc:507] Master 127.18.82.126:43743 requested a full tablet report, sending...
I20251025 20:08:26.772006 23998 ts_manager.cc:194] Registered new tserver with Master: e375d595440549cf985f5fd340f8fe5b (127.18.82.67:37629)
I20251025 20:08:26.778739 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:08:26.781924 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.780506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.18.82.126:43743)
1025 20:08:26.780638 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.780641 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.780754 (+ 113us) server_negotiation.cc:408] Connection header received
1025 20:08:26.780849 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.780851 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.780883 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.780950 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.781659 (+ 709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.781662 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.781664 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.781674 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.781692 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.781699 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.781763 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.781809 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.781835 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:08:26.784313 23998 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:26.786717 23998 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:26.797166 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.794341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:37629 (local address 127.0.0.1:52460)
1025 20:08:26.794713 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.794750 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.794857 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.795746 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.795752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.795818 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.796374 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.796378 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.796415 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.796474 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.796868 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.796876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.796881 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.796984 (+ 103us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.797043 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20251025 20:08:26.797178 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.794510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.18.82.67:37629)
1025 20:08:26.795295 (+ 785us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.795302 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.795325 (+ 23us) server_negotiation.cc:408] Connection header received
1025 20:08:26.795381 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.795389 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.795556 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.796077 (+ 521us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.796724 (+ 647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.796728 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.796735 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.796741 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.796768 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.796776 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.796878 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.797001 (+ 123us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.797060 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":161,"threads_started":1}
I20251025 20:08:26.797446 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.794518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:33267 (local address 127.0.0.1:39848)
1025 20:08:26.794915 (+ 397us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.794930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.795008 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.795922 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.795925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.795943 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.796541 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.796543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.796554 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.796583 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.797233 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.797239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.797243 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.797305 (+ 62us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.797341 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20251025 20:08:26.797984 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.795125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.18.82.65:33267)
1025 20:08:26.795493 (+ 368us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.795503 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.795528 (+ 25us) server_negotiation.cc:408] Connection header received
1025 20:08:26.795603 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.795611 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.795749 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.796131 (+ 382us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.797070 (+ 939us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.797075 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.797082 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.797091 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.797128 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.797138 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.797664 (+ 526us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.797708 (+ 44us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.797808 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20251025 20:08:26.799018 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.795451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37115 (local address 127.0.0.1:59982)
1025 20:08:26.796662 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.796671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.796729 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.797779 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.797782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.797795 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.798369 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.798371 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.798381 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.798402 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.798776 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.798779 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.798782 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.798871 (+ 89us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.798903 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":69,"threads_started":1}
I20251025 20:08:26.799238 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.795816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.18.82.66:37115)
1025 20:08:26.796189 (+ 373us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.796195 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.796745 (+ 550us) server_negotiation.cc:408] Connection header received
1025 20:08:26.796811 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.796819 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.796947 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.797068 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.798528 (+ 1460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.798533 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.798563 (+ 30us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.798575 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.798622 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.798634 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.798735 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.798896 (+ 161us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.798972 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20251025 20:08:26.800009 24382 tablet_service.cc:1505] Processing CreateTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e (DEFAULT_TABLE table=test-table [id=8391b999ffcf4b5bad293db800980e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:26.800562 24120 tablet_service.cc:1505] Processing CreateTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e (DEFAULT_TABLE table=test-table [id=8391b999ffcf4b5bad293db800980e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:26.801050 24382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a71636e5bba47b2a32e0390b7412a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:26.801329 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a71636e5bba47b2a32e0390b7412a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:26.801898 24251 tablet_service.cc:1505] Processing CreateTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e (DEFAULT_TABLE table=test-table [id=8391b999ffcf4b5bad293db800980e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:26.802717 24251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a71636e5bba47b2a32e0390b7412a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:26.806118 24462 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: Bootstrap starting.
I20251025 20:08:26.807935 24464 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Bootstrap starting.
I20251025 20:08:26.808353 24463 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Bootstrap starting.
I20251025 20:08:26.808943 24462 tablet_bootstrap.cc:654] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:26.809717 24462 log.cc:826] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:26.809901 24464 tablet_bootstrap.cc:654] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:26.810263 24463 tablet_bootstrap.cc:654] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:26.810746 24464 log.cc:826] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:26.811007 24463 log.cc:826] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:26.811506 24462 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: No bootstrap required, opened a new log
I20251025 20:08:26.811686 24462 ts_tablet_manager.cc:1403] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 20:08:26.812464 24463 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: No bootstrap required, opened a new log
I20251025 20:08:26.812464 24464 tablet_bootstrap.cc:492] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: No bootstrap required, opened a new log
I20251025 20:08:26.812621 24464 ts_tablet_manager.cc:1403] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:08:26.812621 24463 ts_tablet_manager.cc:1403] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 20:08:26.814023 24462 raft_consensus.cc:359] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.814179 24462 raft_consensus.cc:385] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:26.814229 24462 raft_consensus.cc:740] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380bffbec91f4369af502c0c9f4b2073, State: Initialized, Role: FOLLOWER
I20251025 20:08:26.814607 24462 consensus_queue.cc:260] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.815158 24186 heartbeater.cc:499] Master 127.18.82.126:43743 was elected leader, sending a full tablet report...
I20251025 20:08:26.815352 24462 ts_tablet_manager.cc:1434] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251025 20:08:26.815344 24463 raft_consensus.cc:359] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.815330 24464 raft_consensus.cc:359] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.815526 24463 raft_consensus.cc:385] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:26.815526 24464 raft_consensus.cc:385] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:26.815593 24464 raft_consensus.cc:740] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f481d9a4099949538764fe0700799f79, State: Initialized, Role: FOLLOWER
I20251025 20:08:26.815593 24463 raft_consensus.cc:740] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e375d595440549cf985f5fd340f8fe5b, State: Initialized, Role: FOLLOWER
I20251025 20:08:26.815980 24464 consensus_queue.cc:260] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.815980 24463 consensus_queue.cc:260] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.816704 24464 ts_tablet_manager.cc:1434] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:08:26.816704 24463 ts_tablet_manager.cc:1434] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 20:08:26.817055 24317 heartbeater.cc:499] Master 127.18.82.126:43743 was elected leader, sending a full tablet report...
I20251025 20:08:26.817356 24448 heartbeater.cc:499] Master 127.18.82.126:43743 was elected leader, sending a full tablet report...
I20251025 20:08:26.823311 24469 raft_consensus.cc:493] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:26.823408 24469 raft_consensus.cc:515] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.824352 24469 leader_election.cc:290] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 380bffbec91f4369af502c0c9f4b2073 (127.18.82.65:33267), f481d9a4099949538764fe0700799f79 (127.18.82.66:37115)
I20251025 20:08:26.826071 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.824521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:33267 (local address 127.18.82.67:48631)
1025 20:08:26.824693 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.824707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.824787 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.825027 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.825031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.825045 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.825599 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.825601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.825615 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.825643 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.825893 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.825895 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.825897 (+ 2us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.825952 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.825979 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20251025 20:08:26.826164 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.824561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:48631 (local address 127.18.82.65:33267)
1025 20:08:26.824693 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.824698 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.824766 (+ 68us) server_negotiation.cc:408] Connection header received
1025 20:08:26.824868 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.824871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.824907 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.824990 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.825759 (+ 769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.825763 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.825766 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.825778 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.825799 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.825809 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.825905 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.825963 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.826002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 20:08:26.826622 24140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a71636e5bba47b2a32e0390b7412a5e" candidate_uuid: "e375d595440549cf985f5fd340f8fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380bffbec91f4369af502c0c9f4b2073" is_pre_election: true
I20251025 20:08:26.826858 24140 raft_consensus.cc:2468] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e375d595440549cf985f5fd340f8fe5b in term 0.
I20251025 20:08:26.827086 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.824732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:59337 (local address 127.18.82.66:37115)
1025 20:08:26.824838 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.824841 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.825219 (+ 378us) server_negotiation.cc:408] Connection header received
1025 20:08:26.825310 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.825314 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.825352 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.825423 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.826722 (+ 1299us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.826727 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.826730 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.826743 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.826763 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.826772 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.826861 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.826942 (+ 81us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.826980 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 20:08:26.827131 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.824532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37115 (local address 127.18.82.67:59337)
1025 20:08:26.825123 (+ 591us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.825136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:26.825228 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:26.825443 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:26.825448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:26.825463 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:26.826572 (+ 1109us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:08:26.826573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:26.826582 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:08:26.826602 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:26.826859 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:26.826861 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:26.826864 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:08:26.826929 (+ 65us) client_negotiation.cc:240] Negotiation successful
1025 20:08:26.826967 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":324,"thread_start_us":105,"threads_started":1}
I20251025 20:08:26.827346 24336 leader_election.cc:304] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380bffbec91f4369af502c0c9f4b2073, e375d595440549cf985f5fd340f8fe5b; no voters:
I20251025 20:08:26.827584 24469 raft_consensus.cc:2804] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:26.827643 24469 raft_consensus.cc:493] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:26.827692 24469 raft_consensus.cc:3060] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.827930 24271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a71636e5bba47b2a32e0390b7412a5e" candidate_uuid: "e375d595440549cf985f5fd340f8fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f481d9a4099949538764fe0700799f79" is_pre_election: true
I20251025 20:08:26.828147 24271 raft_consensus.cc:2468] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e375d595440549cf985f5fd340f8fe5b in term 0.
I20251025 20:08:26.828756 24469 raft_consensus.cc:515] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.829015 24469 leader_election.cc:290] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [CANDIDATE]: Term 1 election: Requested vote from peers 380bffbec91f4369af502c0c9f4b2073 (127.18.82.65:33267), f481d9a4099949538764fe0700799f79 (127.18.82.66:37115)
I20251025 20:08:26.829283 24271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a71636e5bba47b2a32e0390b7412a5e" candidate_uuid: "e375d595440549cf985f5fd340f8fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f481d9a4099949538764fe0700799f79"
I20251025 20:08:26.829380 24271 raft_consensus.cc:3060] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.830639 24271 raft_consensus.cc:2468] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e375d595440549cf985f5fd340f8fe5b in term 1.
I20251025 20:08:26.830878 24140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a71636e5bba47b2a32e0390b7412a5e" candidate_uuid: "e375d595440549cf985f5fd340f8fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380bffbec91f4369af502c0c9f4b2073"
I20251025 20:08:26.830924 24337 leader_election.cc:304] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e375d595440549cf985f5fd340f8fe5b, f481d9a4099949538764fe0700799f79; no voters:
I20251025 20:08:26.830999 24140 raft_consensus.cc:3060] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:26.831070 24469 raft_consensus.cc:2804] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:26.831259 24469 raft_consensus.cc:697] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 1 LEADER]: Becoming Leader. State: Replica: e375d595440549cf985f5fd340f8fe5b, State: Running, Role: LEADER
I20251025 20:08:26.831594 24469 consensus_queue.cc:237] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } }
I20251025 20:08:26.832337 24140 raft_consensus.cc:2468] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e375d595440549cf985f5fd340f8fe5b in term 1.
I20251025 20:08:26.834182 23997 catalog_manager.cc:5649] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b reported cstate change: term changed from 0 to 1, leader changed from <none> to e375d595440549cf985f5fd340f8fe5b (127.18.82.67). New cstate: current_term: 1 leader_uuid: "e375d595440549cf985f5fd340f8fe5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e375d595440549cf985f5fd340f8fe5b" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 37629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:26.847867 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.846626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.18.82.67:37629)
1025 20:08:26.846738 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:26.846742 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:26.846753 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:26.846830 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:26.846832 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:26.846880 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:26.846939 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:26.847553 (+ 614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:26.847557 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:26.847558 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:26.847567 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:26.847593 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:26.847599 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:26.847659 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:26.847717 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 20:08:26.847743 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:26.852559 24271 raft_consensus.cc:1275] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 1 FOLLOWER]: Refusing update from remote peer e375d595440549cf985f5fd340f8fe5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:26.852558 24140 raft_consensus.cc:1275] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 1 FOLLOWER]: Refusing update from remote peer e375d595440549cf985f5fd340f8fe5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:26.853070 24469 consensus_queue.cc:1048] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "380bffbec91f4369af502c0c9f4b2073" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:26.853200 24472 consensus_queue.cc:1048] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f481d9a4099949538764fe0700799f79" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:26.860687 24477 mvcc.cc:204] Tried to move back new op lower bound from 7214788226462232576 to 7214788226384572416. Current Snapshot: MvccSnapshot[applied={T|T < 7214788226462232576}]
I20251025 20:08:26.860847 24475 mvcc.cc:204] Tried to move back new op lower bound from 7214788226462232576 to 7214788226384572416. Current Snapshot: MvccSnapshot[applied={T|T < 7214788226462232576}]
I20251025 20:08:26.867905 23998 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:26.868054 23998 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:26.870239 23998 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a3567920a9224d8ab37340a7ce56d0a4: Sending DeleteTablet for 3 replicas of tablet 3a71636e5bba47b2a32e0390b7412a5e
I20251025 20:08:26.870910 24382 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='slave'} at 127.0.0.1:52460
I20251025 20:08:26.870934 24120 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='slave'} at 127.0.0.1:39848
I20251025 20:08:26.871174 24251 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a71636e5bba47b2a32e0390b7412a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:26 UTC) from {username='slave'} at 127.0.0.1:59982
I20251025 20:08:26.871364 24485 tablet_replica.cc:333] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073: stopping tablet replica
I20251025 20:08:26.871639 24485 raft_consensus.cc:2243] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:26.871706 24486 tablet_replica.cc:333] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: stopping tablet replica
I20251025 20:08:26.871887 24486 raft_consensus.cc:2243] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:26.871914 24485 raft_consensus.cc:2272] T 3a71636e5bba47b2a32e0390b7412a5e P 380bffbec91f4369af502c0c9f4b2073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:26.872123 24486 raft_consensus.cc:2272] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:26.872231 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24058
I20251025 20:08:26.873746 24484 tablet_replica.cc:333] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: stopping tablet replica
I20251025 20:08:26.873934 24484 raft_consensus.cc:2243] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:26.874255 24484 raft_consensus.cc:2272] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:26.874348 24486 ts_tablet_manager.cc:1916] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:26.875444 24484 ts_tablet_manager.cc:1916] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:26.876895 24486 ts_tablet_manager.cc:1929] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:26.876986 24486 log.cc:1199] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/3a71636e5bba47b2a32e0390b7412a5e
I20251025 20:08:26.877274 24486 ts_tablet_manager.cc:1950] T 3a71636e5bba47b2a32e0390b7412a5e P f481d9a4099949538764fe0700799f79: Deleting consensus metadata
I20251025 20:08:26.877934 24484 ts_tablet_manager.cc:1929] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:26.878005 24484 log.cc:1199] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/3a71636e5bba47b2a32e0390b7412a5e
I20251025 20:08:26.878300 24484 ts_tablet_manager.cc:1950] T 3a71636e5bba47b2a32e0390b7412a5e P e375d595440549cf985f5fd340f8fe5b: Deleting consensus metadata
I20251025 20:08:26.878449 23984 catalog_manager.cc:4985] TS f481d9a4099949538764fe0700799f79 (127.18.82.66:37115): tablet 3a71636e5bba47b2a32e0390b7412a5e (table test-table [id=8391b999ffcf4b5bad293db800980e28]) successfully deleted
I20251025 20:08:26.879225 23982 catalog_manager.cc:4985] TS e375d595440549cf985f5fd340f8fe5b (127.18.82.67:37629): tablet 3a71636e5bba47b2a32e0390b7412a5e (table test-table [id=8391b999ffcf4b5bad293db800980e28]) successfully deleted
W20251025 20:08:26.882615 23983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:33267 (error 108)
I20251025 20:08:26.882768 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24189
I20251025 20:08:26.884346 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:26.884056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:33267 (local address 127.0.0.1:39858)
1025 20:08:26.884197 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:26.884261 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:33267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20251025 20:08:26.884492 23983 catalog_manager.cc:4712] TS 380bffbec91f4369af502c0c9f4b2073 (127.18.82.65:33267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a71636e5bba47b2a32e0390b7412a5e: Network error: Client connection negotiation failed: client connection to 127.18.82.65:33267: connect: Connection refused (error 111)
I20251025 20:08:26.889060 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24320
I20251025 20:08:26.894661 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 23967
2025-10-25T20:08:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (814 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251025 20:08:26.915508 18761 mini_oidc.cc:150] Starting JWKS server
I20251025 20:08:26.916762 18761 webserver.cc:492] Webserver started at https://127.0.0.1:34785/ using document root <none> and password file <none>
I20251025 20:08:26.916905 18761 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 20:08:26.917098 18761 webserver.cc:492] Webserver started at http://127.0.0.1:44129/ using document root <none> and password file <none>
2025-10-25T20:08:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:26Z Disabled control of system clock
I20251025 20:08:26.924221 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:40933
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:40933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/testchainca.cert with env {}
W20251025 20:08:27.026068 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:27.026254 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:27.026320 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:27.029448 24493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:27.029534 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:27.029552 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:27.029593 24493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:27.029628 24493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 20:08:27.029680 24493 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 20:08:27.029699 24493 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34785/jwks.json
I20251025 20:08:27.033752 24493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:40933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:27.034734 24493 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:27.035535 24493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:27.040995 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:27.040992 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:27.041193 24493 server_base.cc:1047] running on GCE node
W20251025 20:08:27.041009 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:27.041786 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:27.042650 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:27.043797 24493 hybrid_clock.cc:648] HybridClock initialized: now 1761422907043786 us; error 25 us; skew 500 ppm
I20251025 20:08:27.045419 24493 webserver.cc:492] Webserver started at http://127.18.82.126:46453/ using document root <none> and password file <none>
I20251025 20:08:27.045984 24493 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:27.046041 24493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:27.046222 24493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:27.047840 24493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "de379948b5b44b738ffdc29fb54b0e79"
format_stamp: "Formatted at 2025-10-25 20:08:27 on dist-test-slave-d2pk"
server_key: "bf2ad71a35b8a4bf451bf17cf39f955a"
server_key_iv: "de1d8b1e941e15db6cdb3ef748c56a2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:27.048298 24493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "de379948b5b44b738ffdc29fb54b0e79"
format_stamp: "Formatted at 2025-10-25 20:08:27 on dist-test-slave-d2pk"
server_key: "bf2ad71a35b8a4bf451bf17cf39f955a"
server_key_iv: "de1d8b1e941e15db6cdb3ef748c56a2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:27.051488 24493 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:27.053509 24507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:27.054543 24493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:27.054658 24493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "de379948b5b44b738ffdc29fb54b0e79"
format_stamp: "Formatted at 2025-10-25 20:08:27 on dist-test-slave-d2pk"
server_key: "bf2ad71a35b8a4bf451bf17cf39f955a"
server_key_iv: "de1d8b1e941e15db6cdb3ef748c56a2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:27.054736 24493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:27.068197 24493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:27.068776 24493 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:27.068936 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:27.076110 24559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:40933 every 8 connection(s)
I20251025 20:08:27.076115 24493 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:40933
I20251025 20:08:27.077093 24493 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:27.078522 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24493
I20251025 20:08:27.078657 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:27.078917 18761 external_mini_cluster.cc:1468] Setting key 9500fd301f928e956f31db56d9b5bf70
I20251025 20:08:27.080189 24560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:27.085609 24560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: Bootstrap starting.
I20251025 20:08:27.086153 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:27.080000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.18.82.126:40933)
1025 20:08:27.080642 (+ 642us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:27.080654 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:27.080696 (+ 42us) server_negotiation.cc:408] Connection header received
1025 20:08:27.081291 (+ 595us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:27.081306 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:27.081692 (+ 386us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:27.081967 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:27.082474 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.083205 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:27.083901 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.084189 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:27.084671 (+ 482us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:27.084710 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:27.084719 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:27.085052 (+ 333us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:27.085089 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:27.085102 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:27.085198 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:27.085353 (+ 155us) server_negotiation.cc:300] Negotiation successful
1025 20:08:27.085589 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20251025 20:08:27.087978 24560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:27.088773 24560 log.cc:826] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:27.088876 18761 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 20:08:27.090694 24560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: No bootstrap required, opened a new log
I20251025 20:08:27.092952 24560 raft_consensus.cc:359] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } }
I20251025 20:08:27.093130 24560 raft_consensus.cc:385] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:27.093187 24560 raft_consensus.cc:740] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de379948b5b44b738ffdc29fb54b0e79, State: Initialized, Role: FOLLOWER
I20251025 20:08:27.093623 24560 consensus_queue.cc:260] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } }
I20251025 20:08:27.093741 24560 raft_consensus.cc:399] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:27.093802 24560 raft_consensus.cc:493] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:27.093894 24560 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:27.094625 24560 raft_consensus.cc:515] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } }
I20251025 20:08:27.094956 24560 leader_election.cc:304] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de379948b5b44b738ffdc29fb54b0e79; no voters:
I20251025 20:08:27.095220 24560 leader_election.cc:290] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:27.095362 24565 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:27.095547 24565 raft_consensus.cc:697] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [term 1 LEADER]: Becoming Leader. State: Replica: de379948b5b44b738ffdc29fb54b0e79, State: Running, Role: LEADER
I20251025 20:08:27.095814 24565 consensus_queue.cc:237] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } }
I20251025 20:08:27.096378 24560 sys_catalog.cc:565] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:27.097455 24566 sys_catalog.cc:455] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de379948b5b44b738ffdc29fb54b0e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } } }
I20251025 20:08:27.097623 24566 sys_catalog.cc:458] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:27.097466 24567 sys_catalog.cc:455] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de379948b5b44b738ffdc29fb54b0e79. Latest consensus state: current_term: 1 leader_uuid: "de379948b5b44b738ffdc29fb54b0e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de379948b5b44b738ffdc29fb54b0e79" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40933 } } }
I20251025 20:08:27.097958 24567 sys_catalog.cc:458] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79 [sys.catalog]: This master's current role is: LEADER
W20251025 20:08:27.101044 24581 catalog_manager.cc:1568] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:27.101114 24581 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:27.101177 24574 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:27.101949 24574 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:27.107206 24574 catalog_manager.cc:1357] Generated new cluster ID: aeccbce546974d529c49365c96bc87d7
I20251025 20:08:27.107275 24574 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:27.136766 24574 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:27.137696 24574 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:27.143266 24574 catalog_manager.cc:6022] T 00000000000000000000000000000000 P de379948b5b44b738ffdc29fb54b0e79: Generated new TSK 0
I20251025 20:08:27.143931 24574 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:08:27.182297 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:27.160392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.18.82.126:40933)
1025 20:08:27.160528 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:27.160533 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:27.160736 (+ 203us) server_negotiation.cc:408] Connection header received
1025 20:08:27.160943 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:27.160946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:27.160996 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:27.161073 (+ 77us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:27.161560 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.162073 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:27.162995 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.163135 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:27.163217 (+ 82us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:27.181849 (+ 18632us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1025 20:08:27.181936 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:27.182103 (+ 167us) server_negotiation.cc:300] Negotiation successful
1025 20:08:27.182156 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"thread_start_us":118,"threads_started":1}
I20251025 20:08:27.193233 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:27.189951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.18.82.126:40933)
1025 20:08:27.190073 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:27.190076 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:27.190165 (+ 89us) server_negotiation.cc:408] Connection header received
1025 20:08:27.190303 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:27.190305 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:27.190349 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:27.190410 (+ 61us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:27.190803 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.191276 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:27.192066 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.192183 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:27.192219 (+ 36us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:27.192987 (+ 768us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1025 20:08:27.193147 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45428: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":47}
W20251025 20:08:27.193293 24563 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45428: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251025 20:08:27.194166 24596 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:27.189834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40933 (local address 127.0.0.1:45428)
1025 20:08:27.190089 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:27.190103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:27.190203 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:27.190431 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:27.190434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:27.190442 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1025 20:08:27.190697 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:27.190705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.191385 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:27.191389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:27.191970 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:27.191980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:27.192077 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:27.192098 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 20:08:27.193153 (+ 1055us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1025 20:08:27.193304 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.82.126:40933: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20251025 20:08:42.202245 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:42.199224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.18.82.126:40933)
1025 20:08:42.199518 (+ 294us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.199523 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.199539 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:08:42.199626 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.199629 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.199680 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.199746 (+ 66us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:42.200044 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.200570 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.201157 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.201288 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.201345 (+ 57us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:42.201998 (+ 653us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1025 20:08:42.202139 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55466: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
W20251025 20:08:42.202314 24602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55466: JWT verification failed: token expired
W20251025 20:08:42.202360 24601 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:42.199076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40933 (local address 127.0.0.1:55466)
1025 20:08:42.199404 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:42.199422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:42.199527 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:42.199765 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:42.199767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:42.199773 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
1025 20:08:42.199956 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.199961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.200682 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.200685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:42.201072 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.201079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.201167 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.201187 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 20:08:42.202113 (+ 926us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1025 20:08:42.202223 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.82.126:40933: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
W20251025 20:08:42.204746 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:42.203977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.18.82.126:40933)
1025 20:08:42.204098 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.204104 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.204170 (+ 66us) server_negotiation.cc:408] Connection header received
1025 20:08:42.204303 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.204307 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.204352 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.204418 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:42.204648 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55480: BlockingRecv error: recv got EOF from 127.0.0.1:55480 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20251025 20:08:42.208142 24612 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.18.82.126:40933
W20251025 20:08:42.208674 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:42.207844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.18.82.126:40933)
1025 20:08:42.207988 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.207992 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.208092 (+ 100us) server_negotiation.cc:408] Connection header received
1025 20:08:42.208283 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.208287 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.208328 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.208394 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:42.208578 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55490: BlockingRecv error: recv got EOF from 127.0.0.1:55490 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20251025 20:08:42.208832 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24493
2025-10-25T20:08:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251025 20:08:42.249961 18761 mini_oidc.cc:150] Starting JWKS server
I20251025 20:08:42.250862 18761 webserver.cc:492] Webserver started at https://127.0.0.1:42243/ using document root <none> and password file <none>
I20251025 20:08:42.250939 18761 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 20:08:42.251109 18761 webserver.cc:492] Webserver started at http://127.0.0.1:34395/ using document root <none> and password file <none>
2025-10-25T20:08:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:42Z Disabled control of system clock
I20251025 20:08:42.258719 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:39583
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:39583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42243/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/testchainca.cert with env {}
W20251025 20:08:42.360513 24618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:42.360719 24618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:42.360766 24618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:42.363744 24618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:42.363792 24618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:42.363807 24618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:42.363824 24618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:42.363839 24618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 20:08:42.363854 24618 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 20:08:42.363878 24618 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42243/jwks.json
I20251025 20:08:42.367858 24618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:39583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42243/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:42.368752 24618 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:42.369477 24618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:42.375249 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:42.375336 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:42.375289 24618 server_base.cc:1047] running on GCE node
W20251025 20:08:42.375249 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:42.376012 24618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:42.376899 24618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:42.378039 24618 hybrid_clock.cc:648] HybridClock initialized: now 1761422922378025 us; error 32 us; skew 500 ppm
I20251025 20:08:42.379738 24618 webserver.cc:492] Webserver started at http://127.18.82.126:36041/ using document root <none> and password file <none>
I20251025 20:08:42.380261 24618 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:42.380327 24618 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:42.380501 24618 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:42.382153 24618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "eb63d91dd983489a97d2f1f5659d6a31"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "f7a00755a5f3671bd203a3fbcf73a8e1"
server_key_iv: "b41673d2219850f8307082eff2b5a741"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.382651 24618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "eb63d91dd983489a97d2f1f5659d6a31"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "f7a00755a5f3671bd203a3fbcf73a8e1"
server_key_iv: "b41673d2219850f8307082eff2b5a741"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.385738 24618 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 20:08:42.387866 24632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:42.388782 24618 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:42.388887 24618 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "eb63d91dd983489a97d2f1f5659d6a31"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "f7a00755a5f3671bd203a3fbcf73a8e1"
server_key_iv: "b41673d2219850f8307082eff2b5a741"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.388967 24618 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:42.418186 24618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:42.418820 24618 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:42.418978 24618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:42.425550 24618 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:39583
I20251025 20:08:42.425575 24684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:39583 every 8 connection(s)
I20251025 20:08:42.426553 24618 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:42.429243 24685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:42.433579 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24618
I20251025 20:08:42.433702 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:42.433943 18761 external_mini_cluster.cc:1468] Setting key dd8a2d7f8fd94d31f82989d1e55982cb
I20251025 20:08:42.434584 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31: Bootstrap starting.
I20251025 20:08:42.436692 24685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:42.437332 24685 log.cc:826] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:42.438987 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31: No bootstrap required, opened a new log
I20251025 20:08:42.440727 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:42.435046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.18.82.126:39583)
1025 20:08:42.435633 (+ 587us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.435642 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.435673 (+ 31us) server_negotiation.cc:408] Connection header received
1025 20:08:42.436147 (+ 474us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.436162 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.436478 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.436756 (+ 278us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:42.437186 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.437919 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.438545 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.438820 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.439319 (+ 499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:42.439337 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:42.439348 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:42.439698 (+ 350us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:42.439736 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:42.439750 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:42.439842 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:42.440000 (+ 158us) server_negotiation.cc:300] Negotiation successful
1025 20:08:42.440196 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20251025 20:08:42.441387 24685 raft_consensus.cc:359] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } }
I20251025 20:08:42.441624 24685 raft_consensus.cc:385] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:42.441686 24685 raft_consensus.cc:740] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb63d91dd983489a97d2f1f5659d6a31, State: Initialized, Role: FOLLOWER
I20251025 20:08:42.442131 24685 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } }
I20251025 20:08:42.442282 24685 raft_consensus.cc:399] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:42.442350 24685 raft_consensus.cc:493] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:42.442440 24685 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:42.443212 18761 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 20:08:42.443397 24685 raft_consensus.cc:515] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } }
I20251025 20:08:42.443763 24685 leader_election.cc:304] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb63d91dd983489a97d2f1f5659d6a31; no voters:
I20251025 20:08:42.444051 24685 leader_election.cc:290] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:42.444141 24690 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:42.444365 24690 raft_consensus.cc:697] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [term 1 LEADER]: Becoming Leader. State: Replica: eb63d91dd983489a97d2f1f5659d6a31, State: Running, Role: LEADER
I20251025 20:08:42.444675 24690 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } }
I20251025 20:08:42.445173 24685 sys_catalog.cc:565] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:42.445811 24692 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb63d91dd983489a97d2f1f5659d6a31. Latest consensus state: current_term: 1 leader_uuid: "eb63d91dd983489a97d2f1f5659d6a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } } }
I20251025 20:08:42.445987 24692 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:42.446151 24691 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb63d91dd983489a97d2f1f5659d6a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63d91dd983489a97d2f1f5659d6a31" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 39583 } } }
I20251025 20:08:42.446269 24691 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:42.446455 24699 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:42.449286 24699 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:42.454619 24699 catalog_manager.cc:1357] Generated new cluster ID: 8ebba96f5ff744c5ae72cd1d50ef1d96
I20251025 20:08:42.454674 24699 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:42.474581 24699 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:42.475696 24699 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:42.484565 24699 catalog_manager.cc:6022] T 00000000000000000000000000000000 P eb63d91dd983489a97d2f1f5659d6a31: Generated new TSK 0
I20251025 20:08:42.485329 24699 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251025 20:08:42.519212 24715 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251025 20:08:42.519297 24715 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251025 20:08:42.520009 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:42.500845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.18.82.126:39583)
1025 20:08:42.500980 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.500983 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.501087 (+ 104us) server_negotiation.cc:408] Connection header received
1025 20:08:42.501242 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.501244 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.501292 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.501369 (+ 77us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:42.501859 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.502358 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.503248 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.503391 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.503474 (+ 83us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:42.519587 (+ 16113us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 20:08:42.519876 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20251025 20:08:42.520087 24688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251025 20:08:42.520079 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:42.500728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:39583 (local address 127.0.0.1:46248)
1025 20:08:42.501008 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:42.501023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:42.501130 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:42.501386 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:42.501389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:42.501396 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1025 20:08:42.501714 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.501724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.502474 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.502478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:42.503130 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.503141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.503250 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.503273 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 20:08:42.519783 (+ 16510us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 20:08:42.519922 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.82.126:39583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20251025 20:08:42.520695 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24618
2025-10-25T20:08:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251025 20:08:42.560230 18761 mini_oidc.cc:150] Starting JWKS server
I20251025 20:08:42.561062 18761 webserver.cc:492] Webserver started at https://127.0.0.1:34235/ using document root <none> and password file <none>
I20251025 20:08:42.561151 18761 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 20:08:42.561378 18761 webserver.cc:492] Webserver started at http://127.0.0.1:46191/ using document root <none> and password file <none>
2025-10-25T20:08:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:42Z Disabled control of system clock
I20251025 20:08:42.569092 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44243
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:44243
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34235/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251025 20:08:42.669157 24721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:42.669368 24721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:42.669440 24721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:42.672480 24721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:42.672536 24721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:42.672565 24721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:42.672593 24721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:42.672632 24721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 20:08:42.672673 24721 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 20:08:42.672701 24721 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34235/jwks.json
I20251025 20:08:42.676055 24721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:44243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34235/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:42.677067 24721 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:42.678105 24721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:42.683358 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:42.683333 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:42.683413 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:42.683537 24721 server_base.cc:1047] running on GCE node
I20251025 20:08:42.684078 24721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:42.684918 24721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:42.686064 24721 hybrid_clock.cc:648] HybridClock initialized: now 1761422922686051 us; error 24 us; skew 500 ppm
I20251025 20:08:42.688108 24721 webserver.cc:492] Webserver started at http://127.18.82.126:41363/ using document root <none> and password file <none>
I20251025 20:08:42.688823 24721 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:42.688884 24721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:42.689095 24721 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:42.690748 24721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "a489da85c50a4ae287a8be6674e46102"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "fb8c22c18818509b7f5f8c974dededf1"
server_key_iv: "e72edaabe896f293b051ff7188c70167"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.691215 24721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "a489da85c50a4ae287a8be6674e46102"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "fb8c22c18818509b7f5f8c974dededf1"
server_key_iv: "e72edaabe896f293b051ff7188c70167"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.694633 24721 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:42.696987 24735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:42.697958 24721 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:42.698060 24721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "a489da85c50a4ae287a8be6674e46102"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "fb8c22c18818509b7f5f8c974dededf1"
server_key_iv: "e72edaabe896f293b051ff7188c70167"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.698139 24721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:42.709645 24721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:42.710201 24721 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:42.710356 24721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:42.717013 24721 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:44243
I20251025 20:08:42.717017 24787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:44243 every 8 connection(s)
I20251025 20:08:42.718010 24721 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:42.720693 24788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:42.723631 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24721
I20251025 20:08:42.723743 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:42.723982 18761 external_mini_cluster.cc:1468] Setting key d1a608eba2327ab15575a6bd67c7c7db
I20251025 20:08:42.727152 24788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102: Bootstrap starting.
I20251025 20:08:42.729218 24788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:42.730088 24788 log.cc:826] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:42.731293 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:42.725079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.18.82.126:44243)
1025 20:08:42.725792 (+ 713us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.725801 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.725833 (+ 32us) server_negotiation.cc:408] Connection header received
1025 20:08:42.726379 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.726397 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.726733 (+ 336us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.727032 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:42.727515 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.728206 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.728814 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.729074 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.729636 (+ 562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:42.729656 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:42.729667 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:42.730107 (+ 440us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:42.730159 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:42.730178 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:42.730302 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:42.730451 (+ 149us) server_negotiation.cc:300] Negotiation successful
1025 20:08:42.730660 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":102,"threads_started":1}
I20251025 20:08:42.732100 24788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102: No bootstrap required, opened a new log
I20251025 20:08:42.733927 18761 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 20:08:42.734901 24788 raft_consensus.cc:359] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } }
I20251025 20:08:42.735106 24788 raft_consensus.cc:385] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:42.735172 24788 raft_consensus.cc:740] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a489da85c50a4ae287a8be6674e46102, State: Initialized, Role: FOLLOWER
I20251025 20:08:42.735597 24788 consensus_queue.cc:260] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } }
I20251025 20:08:42.735738 24788 raft_consensus.cc:399] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:42.735798 24788 raft_consensus.cc:493] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:42.735891 24788 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:42.736657 24788 raft_consensus.cc:515] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } }
I20251025 20:08:42.736970 24788 leader_election.cc:304] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a489da85c50a4ae287a8be6674e46102; no voters:
I20251025 20:08:42.737221 24788 leader_election.cc:290] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:42.737313 24793 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:42.737555 24793 raft_consensus.cc:697] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [term 1 LEADER]: Becoming Leader. State: Replica: a489da85c50a4ae287a8be6674e46102, State: Running, Role: LEADER
I20251025 20:08:42.737829 24793 consensus_queue.cc:237] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } }
I20251025 20:08:42.738257 24788 sys_catalog.cc:565] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:42.739311 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a489da85c50a4ae287a8be6674e46102. Latest consensus state: current_term: 1 leader_uuid: "a489da85c50a4ae287a8be6674e46102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } } }
I20251025 20:08:42.739467 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:42.739873 24802 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:42.739311 24794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a489da85c50a4ae287a8be6674e46102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a489da85c50a4ae287a8be6674e46102" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44243 } } }
I20251025 20:08:42.740054 24794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:42.742789 24802 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:42.747650 24802 catalog_manager.cc:1357] Generated new cluster ID: e0844b9be4504e459c3c53c0f63b6c04
I20251025 20:08:42.747718 24802 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:42.766615 24802 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:42.767815 24802 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:42.775595 24802 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a489da85c50a4ae287a8be6674e46102: Generated new TSK 0
I20251025 20:08:42.776373 24802 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251025 20:08:42.817946 24818 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251025 20:08:42.818022 24818 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251025 20:08:42.819186 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:42.795914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.18.82.126:44243)
1025 20:08:42.796059 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:42.796064 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:42.796166 (+ 102us) server_negotiation.cc:408] Connection header received
1025 20:08:42.796335 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:42.796337 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:42.796385 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:42.796466 (+ 81us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:42.796889 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.797388 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.798260 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.798396 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.798481 (+ 85us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:42.818798 (+ 20317us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 20:08:42.819071 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20251025 20:08:42.819243 24791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251025 20:08:42.819329 24817 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:42.795789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44243 (local address 127.0.0.1:33336)
1025 20:08:42.796085 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:42.796100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:42.796216 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:42.796484 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:42.796488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:42.796495 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1025 20:08:42.796797 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.796806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.797503 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:42.797531 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:42.798156 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:42.798167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:42.798277 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:42.798299 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 20:08:42.818990 (+ 20691us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 20:08:42.819151 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.82.126:44243: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
I20251025 20:08:42.819929 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24721
2025-10-25T20:08:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-25T20:08:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:42Z Disabled control of system clock
I20251025 20:08:42.867518 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35003
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:35003
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/testchainca.cert with env {}
W20251025 20:08:42.966032 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:42.966236 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:42.966280 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:42.969154 24822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:42.969197 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:42.969210 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:42.969244 24822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:42.969259 24822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 20:08:42.969274 24822 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 20:08:42.969287 24822 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251025 20:08:42.973239 24822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:35003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:42.974192 24822 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:42.974938 24822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:42.980073 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:42.980080 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:42.980135 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:42.980353 24822 server_base.cc:1047] running on GCE node
I20251025 20:08:42.980801 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:42.981702 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:42.982851 24822 hybrid_clock.cc:648] HybridClock initialized: now 1761422922982846 us; error 41 us; skew 500 ppm
I20251025 20:08:42.984493 24822 webserver.cc:492] Webserver started at http://127.18.82.126:34535/ using document root <none> and password file <none>
I20251025 20:08:42.985033 24822 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:42.985095 24822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:42.985265 24822 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:42.986961 24822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "a315e4f05a594e03b76f5c93b24ddd82"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "67b1e538e2152ba7d19909a3d1e06247"
server_key_iv: "796e6e907ccecd36f1052275ed558122"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.987439 24822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "a315e4f05a594e03b76f5c93b24ddd82"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "67b1e538e2152ba7d19909a3d1e06247"
server_key_iv: "796e6e907ccecd36f1052275ed558122"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.990556 24822 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:42.992578 24836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:42.993479 24822 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:42.993621 24822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "a315e4f05a594e03b76f5c93b24ddd82"
format_stamp: "Formatted at 2025-10-25 20:08:42 on dist-test-slave-d2pk"
server_key: "67b1e538e2152ba7d19909a3d1e06247"
server_key_iv: "796e6e907ccecd36f1052275ed558122"
server_key_version: "encryptionkey@0"
I20251025 20:08:42.993703 24822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:43.003116 24822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:43.003651 24822 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:43.003831 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:43.010576 24822 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:35003
I20251025 20:08:43.010573 24888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:35003 every 8 connection(s)
I20251025 20:08:43.011533 24822 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:43.011894 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24822
I20251025 20:08:43.012022 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:43.012281 18761 external_mini_cluster.cc:1468] Setting key 4d9bcf12c83f018dfbb32389fbca486d
I20251025 20:08:43.014840 24889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:43.020622 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:43.013500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.18.82.126:35003)
1025 20:08:43.014185 (+ 685us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:43.014195 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:43.014233 (+ 38us) server_negotiation.cc:408] Connection header received
1025 20:08:43.015031 (+ 798us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:43.015055 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:43.015452 (+ 397us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:43.015783 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:43.016235 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.017816 (+ 1581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:43.018420 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.018676 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:43.019058 (+ 382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:43.019101 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:43.019112 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:08:43.019514 (+ 402us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:43.019563 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:08:43.019579 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:43.019683 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:43.019783 (+ 100us) server_negotiation.cc:300] Negotiation successful
1025 20:08:43.019992 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20251025 20:08:43.020639 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82: Bootstrap starting.
I20251025 20:08:43.022758 24889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:43.023111 18761 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 20:08:43.023473 24889 log.cc:826] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:43.025064 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82: No bootstrap required, opened a new log
I20251025 20:08:43.027501 24889 raft_consensus.cc:359] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } }
I20251025 20:08:43.027679 24889 raft_consensus.cc:385] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:43.027741 24889 raft_consensus.cc:740] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a315e4f05a594e03b76f5c93b24ddd82, State: Initialized, Role: FOLLOWER
I20251025 20:08:43.028133 24889 consensus_queue.cc:260] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } }
I20251025 20:08:43.028254 24889 raft_consensus.cc:399] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:43.028314 24889 raft_consensus.cc:493] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:43.028406 24889 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:43.029143 24889 raft_consensus.cc:515] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } }
I20251025 20:08:43.029450 24889 leader_election.cc:304] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a315e4f05a594e03b76f5c93b24ddd82; no voters:
I20251025 20:08:43.029736 24889 leader_election.cc:290] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:43.029834 24894 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:43.030025 24894 raft_consensus.cc:697] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [term 1 LEADER]: Becoming Leader. State: Replica: a315e4f05a594e03b76f5c93b24ddd82, State: Running, Role: LEADER
I20251025 20:08:43.030277 24894 consensus_queue.cc:237] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } }
I20251025 20:08:43.030697 24889 sys_catalog.cc:565] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:43.031718 24895 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a315e4f05a594e03b76f5c93b24ddd82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } } }
I20251025 20:08:43.031844 24895 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:43.032248 24903 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:43.032763 24896 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a315e4f05a594e03b76f5c93b24ddd82. Latest consensus state: current_term: 1 leader_uuid: "a315e4f05a594e03b76f5c93b24ddd82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315e4f05a594e03b76f5c93b24ddd82" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35003 } } }
I20251025 20:08:43.032871 24896 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:43.034821 24903 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:43.039633 24903 catalog_manager.cc:1357] Generated new cluster ID: aaa7132504c94531b563db38cba5c48f
I20251025 20:08:43.039695 24903 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:43.045670 24903 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:43.046741 24903 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:43.053072 24903 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a315e4f05a594e03b76f5c93b24ddd82: Generated new TSK 0
I20251025 20:08:43.053705 24903 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:08:43.132841 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:43.066367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.18.82.126:35003)
1025 20:08:43.066535 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:43.066539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:43.066686 (+ 147us) server_negotiation.cc:408] Connection header received
1025 20:08:43.066838 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:43.066841 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:43.066889 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:43.066966 (+ 77us) server_negotiation.cc:227] Negotiated authn=JWT
1025 20:08:43.067393 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.067873 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:43.068626 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.068735 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:43.068804 (+ 69us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 20:08:43.132435 (+ 63631us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1025 20:08:43.132703 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36900: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20251025 20:08:43.132911 24892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36900: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251025 20:08:43.133028 24918 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:43.066277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35003 (local address 127.0.0.1:36900)
1025 20:08:43.066601 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:43.066617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:43.066726 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:43.066999 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:43.067003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:43.067011 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1025 20:08:43.067293 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:43.067301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.067968 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:43.067973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:43.068538 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:43.068549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:43.068659 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:43.068679 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 20:08:43.132658 (+ 63979us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1025 20:08:43.132839 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.82.126:35003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":109,"threads_started":1}
I20251025 20:08:43.133741 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24822
2025-10-25T20:08:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (292 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (108 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (108 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:43 dist-test-slave-d2pk krb5kdc[24930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:43 dist-test-slave-d2pk krb5kdc[24930](info): set up 2 sockets
Oct 25 20:08:43 dist-test-slave-d2pk krb5kdc[24930](info): commencing operation
krb5kdc: starting...
W20251025 20:08:45.186220 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:45.309829 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35853
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:45.409226 24946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:45.409405 24946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:45.409454 24946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:45.412489 24946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:45.412542 24946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:45.412565 24946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:45.412595 24946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:45.412612 24946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:45.416159 24946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:35853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:45.417042 24946 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:45.417780 24946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:45.422861 24952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:45.422861 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:45.422857 24954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:45.423012 24946 server_base.cc:1047] running on GCE node
I20251025 20:08:45.423453 24946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:45.424242 24946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:45.425379 24946 hybrid_clock.cc:648] HybridClock initialized: now 1761422925425372 us; error 25 us; skew 500 ppm
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:45.427714 24946 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:08:45.428593 24946 webserver.cc:492] Webserver started at http://127.18.82.126:41485/ using document root <none> and password file <none>
I20251025 20:08:45.429036 24946 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:45.429075 24946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:45.429203 24946 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:45.430727 24946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "1c4b9c4975d347b39a1e1bd481f512d0"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "149b0821ae029f60cc1bc1ebf91635bb"
server_key_iv: "251496fde9c485511e5aee64f7adfb09"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.431108 24946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "1c4b9c4975d347b39a1e1bd481f512d0"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "149b0821ae029f60cc1bc1ebf91635bb"
server_key_iv: "251496fde9c485511e5aee64f7adfb09"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.434098 24946 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:45.435920 24961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.436724 24946 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:45.436802 24946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "1c4b9c4975d347b39a1e1bd481f512d0"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "149b0821ae029f60cc1bc1ebf91635bb"
server_key_iv: "251496fde9c485511e5aee64f7adfb09"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.436864 24946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:45.447717 24946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:45.449854 24946 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:45.449970 24946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:45.456048 25013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:35853 every 8 connection(s)
I20251025 20:08:45.456054 24946 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:35853
I20251025 20:08:45.456880 24946 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:45.459129 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:45.463975 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0: Bootstrap starting.
I20251025 20:08:45.464906 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 24946
I20251025 20:08:45.465031 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:45.465263 18761 external_mini_cluster.cc:1468] Setting key 3eb1220b8428b54ae631ebc1d33c1f91
I20251025 20:08:45.466480 25014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:45.467133 25014 log.cc:826] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:45.469062 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0: No bootstrap required, opened a new log
I20251025 20:08:45.472061 25014 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } }
I20251025 20:08:45.472239 25014 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:45.472278 25014 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4b9c4975d347b39a1e1bd481f512d0, State: Initialized, Role: FOLLOWER
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:45.472702 25014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } }
I20251025 20:08:45.472829 25014 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:45.472894 25014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:45.472985 25014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:45.474013 25014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } }
I20251025 20:08:45.474411 25014 leader_election.cc:304] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4b9c4975d347b39a1e1bd481f512d0; no voters:
I20251025 20:08:45.474695 25014 leader_election.cc:290] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:45.474776 25019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:45.474965 25019 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [term 1 LEADER]: Becoming Leader. State: Replica: 1c4b9c4975d347b39a1e1bd481f512d0, State: Running, Role: LEADER
I20251025 20:08:45.475276 25019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } }
I20251025 20:08:45.475770 25014 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:45.476768 25020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } } }
I20251025 20:08:45.476861 25020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:45.477208 25028 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:45.477979 25021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4b9c4975d347b39a1e1bd481f512d0. Latest consensus state: current_term: 1 leader_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b9c4975d347b39a1e1bd481f512d0" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35853 } } }
I20251025 20:08:45.478081 25021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:45.478644 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:45.466249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.18.82.126:35853)
1025 20:08:45.466704 (+ 455us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:45.466713 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:45.466744 (+ 31us) server_negotiation.cc:408] Connection header received
1025 20:08:45.467241 (+ 497us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:45.467266 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:45.467595 (+ 329us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:45.467915 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:45.468851 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.469609 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:45.470361 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.470552 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:45.472858 (+ 2306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:45.472880 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:45.472893 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:45.472937 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:45.475399 (+ 2462us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.475874 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.475882 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.475889 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.475973 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.476220 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.476223 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.476225 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.476625 (+ 400us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:45.476826 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:45.477740 (+ 914us) server_negotiation.cc:300] Negotiation successful
1025 20:08:45.477971 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20251025 20:08:45.480099 25028 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:45.484804 25028 catalog_manager.cc:1357] Generated new cluster ID: ec771ebeea424c21a9547af45b9a707a
I20251025 20:08:45.484866 25028 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:45.541406 25028 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:45.542289 25028 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:45.553021 25028 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1c4b9c4975d347b39a1e1bd481f512d0: Generated new TSK 0
I20251025 20:08:45.553747 25028 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:45.602234 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35853
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:45.696530 25042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:45.696698 25042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:45.696746 25042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:45.699708 25042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:45.699751 25042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:45.699815 25042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:45.703444 25042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:45.704193 25042 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:45.704988 25042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:45.711463 25048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:45.711485 25047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:45.711602 25042 server_base.cc:1047] running on GCE node
W20251025 20:08:45.711692 25050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:45.712015 25042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:45.712492 25042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:45.713632 25042 hybrid_clock.cc:648] HybridClock initialized: now 1761422925713612 us; error 35 us; skew 500 ppm
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:45.715909 25042 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:08:45.716907 25042 webserver.cc:492] Webserver started at http://127.18.82.65:35757/ using document root <none> and password file <none>
I20251025 20:08:45.717371 25042 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:45.717415 25042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:45.717633 25042 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:45.719297 25042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "e6fd39df767b4aa2b3de8dd25d6ee5ce"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "861343d7f81c1730224c41baf99bb1f1"
server_key_iv: "1e44a24372a05ab092089a0a5d56e57e"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.719702 25042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "e6fd39df767b4aa2b3de8dd25d6ee5ce"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "861343d7f81c1730224c41baf99bb1f1"
server_key_iv: "1e44a24372a05ab092089a0a5d56e57e"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.722749 25042 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:08:45.724725 25057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.725610 25042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:08:45.725701 25042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "e6fd39df767b4aa2b3de8dd25d6ee5ce"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "861343d7f81c1730224c41baf99bb1f1"
server_key_iv: "1e44a24372a05ab092089a0a5d56e57e"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.725777 25042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:45.745935 25042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:45.751755 25042 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:45.751895 25042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:45.752379 25042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:45.753157 25042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:45.753206 25042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.753260 25042 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:45.753297 25042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.760987 25042 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:38901
I20251025 20:08:45.761006 25170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:38901 every 8 connection(s)
I20251025 20:08:45.761773 25042 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:45.766623 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25042
I20251025 20:08:45.766734 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:45.766949 18761 external_mini_cluster.cc:1468] Setting key ac3969fdd2363d1a08666b90d3b19bdb
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:45.773241 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:45.763308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:54553 (local address 127.18.82.126:35853)
1025 20:08:45.763458 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:45.763462 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:45.764066 (+ 604us) server_negotiation.cc:408] Connection header received
1025 20:08:45.764788 (+ 722us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:45.764792 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:45.764847 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:45.764913 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:45.766384 (+ 1471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.766919 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:45.767576 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.767765 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:45.770132 (+ 2367us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:45.770150 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:45.770153 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:45.770174 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:45.771536 (+ 1362us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.771949 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.771952 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.771953 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.771992 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.772251 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.772255 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.772257 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.772410 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:45.772502 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:45.772959 (+ 457us) server_negotiation.cc:300] Negotiation successful
1025 20:08:45.773083 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 20:08:45.773867 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:45.763546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35853 (local address 127.18.82.65:54553)
1025 20:08:45.763954 (+ 408us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:45.763984 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:45.764622 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:45.765047 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:45.765054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:45.765353 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:45.766212 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:45.766224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.767046 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:45.767050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:45.767440 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:45.767447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.767665 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:45.768256 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:45.768316 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:45.769952 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:45.771642 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:45.771647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:45.771657 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:45.771860 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:45.772066 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:45.772070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:45.772073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:45.772167 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:45.772513 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:45.772521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:45.772733 (+ 212us) client_negotiation.cc:769] Sending connection context
1025 20:08:45.772909 (+ 176us) client_negotiation.cc:240] Negotiation successful
1025 20:08:45.773160 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20251025 20:08:45.774919 25171 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35853
I20251025 20:08:45.775156 25171 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
I20251025 20:08:45.775727 25171 heartbeater.cc:507] Master 127.18.82.126:35853 requested a full tablet report, sending...
I20251025 20:08:45.776983 24978 ts_manager.cc:194] Registered new tserver with Master: e6fd39df767b4aa2b3de8dd25d6ee5ce (127.18.82.65:38901)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
I20251025 20:08:45.778247 24978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:54553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:45.808331 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35853
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:45.904938 25178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:45.905115 25178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:45.905176 25178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:45.908340 25178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:45.908403 25178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:45.908490 25178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:45.912371 25178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:45.913260 25178 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:45.914045 25178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:45.919751 25184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:45.919777 25186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:45.919785 25183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:45.919911 25178 server_base.cc:1047] running on GCE node
I20251025 20:08:45.920303 25178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:45.920779 25178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:45.921922 25178 hybrid_clock.cc:648] HybridClock initialized: now 1761422925921898 us; error 39 us; skew 500 ppm
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:45.924242 25178 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:08:45.925122 25178 webserver.cc:492] Webserver started at http://127.18.82.66:35173/ using document root <none> and password file <none>
I20251025 20:08:45.925668 25178 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:45.925720 25178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:45.925894 25178 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:45.927436 25178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "c616f1caf0b8435da247f235e473ad4d"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "4c6c9157967c9e48d2b1db27ad8520e2"
server_key_iv: "4cc25ac91755924f437618bfa67d96fc"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.927886 25178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "c616f1caf0b8435da247f235e473ad4d"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "4c6c9157967c9e48d2b1db27ad8520e2"
server_key_iv: "4cc25ac91755924f437618bfa67d96fc"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.930852 25178 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:45.932732 25193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.933676 25178 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:08:45.933768 25178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "c616f1caf0b8435da247f235e473ad4d"
format_stamp: "Formatted at 2025-10-25 20:08:45 on dist-test-slave-d2pk"
server_key: "4c6c9157967c9e48d2b1db27ad8520e2"
server_key_iv: "4cc25ac91755924f437618bfa67d96fc"
server_key_version: "encryptionkey@0"
I20251025 20:08:45.933868 25178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:45.944681 25178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:45.947067 25178 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:45.947219 25178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:45.947726 25178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:45.948462 25178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:45.948525 25178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.948609 25178 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:45.948642 25178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:45.957000 25178 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:42807
I20251025 20:08:45.957022 25306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:42807 every 8 connection(s)
I20251025 20:08:45.957777 25178 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:45.962620 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25178
I20251025 20:08:45.962724 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:45.962973 18761 external_mini_cluster.cc:1468] Setting key 6646bb7dbc56b462f89bf10d87af0ac8
Oct 25 20:08:45 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422925, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:45.969048 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:45.959275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:50371 (local address 127.18.82.126:35853)
1025 20:08:45.959440 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:45.959444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:45.960022 (+ 578us) server_negotiation.cc:408] Connection header received
1025 20:08:45.960684 (+ 662us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:45.960687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:45.960731 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:45.960797 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:45.962107 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.962604 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:45.963447 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.963605 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:45.965759 (+ 2154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:45.965774 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:45.965776 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:45.965797 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:45.967357 (+ 1560us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.967790 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.967796 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.967801 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.967852 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:45.968097 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:45.968101 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:45.968103 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:45.968233 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:45.968304 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:45.968861 (+ 557us) server_negotiation.cc:300] Negotiation successful
1025 20:08:45.968944 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:08:45.970005 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:45.959534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35853 (local address 127.18.82.66:50371)
1025 20:08:45.959896 (+ 362us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:45.959927 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:45.960528 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:45.960911 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:45.960917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:45.961210 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:45.962000 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:45.962012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.962725 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:45.962731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:45.963283 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:45.963295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:45.963491 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:45.964027 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:45.964047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:45.965630 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:45.967476 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:45.967480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:45.967488 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:45.967704 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:45.967926 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:45.967928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:45.967929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:45.967988 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:45.968335 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:45.968341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:45.968622 (+ 281us) client_negotiation.cc:769] Sending connection context
1025 20:08:45.968851 (+ 229us) client_negotiation.cc:240] Negotiation successful
1025 20:08:45.969111 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20251025 20:08:45.971037 25307 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35853
I20251025 20:08:45.971259 25307 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:45.971693 25307 heartbeater.cc:507] Master 127.18.82.126:35853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:08:45.972594 24978 ts_manager.cc:194] Registered new tserver with Master: c616f1caf0b8435da247f235e473ad4d (127.18.82.66:42807)
I20251025 20:08:45.973130 24978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:50371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:08:46.004350 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35853
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:46.101284 25314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:46.101496 25314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:46.101585 25314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:46.104528 25314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:46.104580 25314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:46.104669 25314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:46.108449 25314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:46.109323 25314 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:46.110108 25314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:46.116142 25322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:46.116196 25314 server_base.cc:1047] running on GCE node
W20251025 20:08:46.116215 25319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:46.116137 25320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:46.116784 25314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:46.117249 25314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:46.118388 25314 hybrid_clock.cc:648] HybridClock initialized: now 1761422926118374 us; error 29 us; skew 500 ppm
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.120553 25314 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:08:46.121393 25314 webserver.cc:492] Webserver started at http://127.18.82.67:38021/ using document root <none> and password file <none>
I20251025 20:08:46.121886 25314 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:46.121940 25314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:46.122114 25314 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:46.123746 25314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "25b526fb70cc4ecbb37d1531deedd9b4"
format_stamp: "Formatted at 2025-10-25 20:08:46 on dist-test-slave-d2pk"
server_key: "67d7e618215aa14cf613655ddb4aa3e3"
server_key_iv: "7384d015a82fd507ccc50ea3b01771b5"
server_key_version: "encryptionkey@0"
I20251025 20:08:46.124168 25314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "25b526fb70cc4ecbb37d1531deedd9b4"
format_stamp: "Formatted at 2025-10-25 20:08:46 on dist-test-slave-d2pk"
server_key: "67d7e618215aa14cf613655ddb4aa3e3"
server_key_iv: "7384d015a82fd507ccc50ea3b01771b5"
server_key_version: "encryptionkey@0"
I20251025 20:08:46.127243 25314 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:46.129253 25329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:46.130245 25314 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:46.130349 25314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "25b526fb70cc4ecbb37d1531deedd9b4"
format_stamp: "Formatted at 2025-10-25 20:08:46 on dist-test-slave-d2pk"
server_key: "67d7e618215aa14cf613655ddb4aa3e3"
server_key_iv: "7384d015a82fd507ccc50ea3b01771b5"
server_key_version: "encryptionkey@0"
I20251025 20:08:46.130452 25314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:46.147097 25314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:46.149871 25314 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:46.150017 25314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:46.150468 25314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:46.151253 25314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:46.151298 25314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:46.151336 25314 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:46.151356 25314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:46.159513 25314 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:46681
I20251025 20:08:46.159548 25442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:46681 every 8 connection(s)
I20251025 20:08:46.160425 25314 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:08:46.168884 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25314
I20251025 20:08:46.169000 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:46.169239 18761 external_mini_cluster.cc:1468] Setting key 4dfdcc320b708b66dc394f77f16089c9
I20251025 20:08:46.171753 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:46.162024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:51757 (local address 127.18.82.126:35853)
1025 20:08:46.162170 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.162200 (+ 30us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.162824 (+ 624us) server_negotiation.cc:408] Connection header received
1025 20:08:46.163700 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.163704 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.163766 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.163836 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.165333 (+ 1497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.165826 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:46.166463 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.166651 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:46.168578 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:46.168601 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:46.168604 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:46.168626 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:46.170141 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.170546 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.170549 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.170551 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.170599 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.170866 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.170869 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.170870 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.170998 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:46.171084 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:46.171548 (+ 464us) server_negotiation.cc:300] Negotiation successful
1025 20:08:46.171646 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251025 20:08:46.172333 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:46.162271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35853 (local address 127.18.82.67:51757)
1025 20:08:46.162687 (+ 416us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:46.162722 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:46.163474 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:46.163989 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:46.163996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:46.164327 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:46.165190 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:46.165204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.165942 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:46.165945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:46.166319 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:46.166326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.166557 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:46.167118 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:46.167140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:46.168447 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:46.170236 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:46.170241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:46.170248 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:46.170459 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:46.170678 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:46.170681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:46.170683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:46.170783 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:46.171092 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:46.171098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:46.171333 (+ 235us) client_negotiation.cc:769] Sending connection context
1025 20:08:46.171496 (+ 163us) client_negotiation.cc:240] Negotiation successful
1025 20:08:46.171685 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20251025 20:08:46.173305 25443 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35853
I20251025 20:08:46.173561 25443 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:46.174014 25443 heartbeater.cc:507] Master 127.18.82.126:35853 requested a full tablet report, sending...
I20251025 20:08:46.174964 24978 ts_manager.cc:194] Registered new tserver with Master: 25b526fb70cc4ecbb37d1531deedd9b4 (127.18.82.67:46681)
I20251025 20:08:46.175468 24978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:51757
I20251025 20:08:46.182876 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/test-admin.keytab.
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.197589 18761 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251025 20:08:46.200021 25450 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:46.198779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.18.82.65:38901)
1025 20:08:46.199072 (+ 293us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.199078 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.199098 (+ 20us) server_negotiation.cc:408] Connection header received
1025 20:08:46.199144 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.199149 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.199298 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.199432 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.199863 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41174: BlockingRecv error: recv got EOF from 127.0.0.1:41174 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
W20251025 20:08:46.200181 18761 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc)
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.201100 18761 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.201974 18761 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:08:46.210222 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:46.202806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.18.82.65:38901)
1025 20:08:46.202932 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.202936 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.202950 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:08:46.203046 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.203049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.203090 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.203156 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.203738 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.204383 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:46.204968 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.205147 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:46.207059 (+ 1912us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:46.207082 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:46.207094 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:46.207118 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:46.208718 (+ 1600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.209107 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.209112 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.209114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.209161 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.209371 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.209375 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.209376 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.209590 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:46.209787 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:46.209957 (+ 170us) server_negotiation.cc:300] Negotiation successful
1025 20:08:46.210089 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20251025 20:08:46.213554 25450 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:46.212552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.18.82.65:38901)
1025 20:08:46.212675 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.212679 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.212831 (+ 152us) server_negotiation.cc:408] Connection header received
1025 20:08:46.212934 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.212938 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.212978 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.213048 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.213445 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41188: BlockingRecv error: recv got EOF from 127.0.0.1:41188 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20251025 20:08:46.213721 18761 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc' of type FILE
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.214641 18761 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.215479 18761 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:08:46.222076 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:46.216216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.18.82.65:38901)
1025 20:08:46.216332 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.216335 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.216349 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:08:46.216441 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.216443 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.216474 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.216532 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.217026 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.217460 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:46.218002 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.218133 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:46.219811 (+ 1678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:46.219825 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:46.219827 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:46.219847 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:46.220914 (+ 1067us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.221202 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.221204 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.221205 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.221234 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.221370 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.221371 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.221372 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.221467 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:46.221646 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:46.221776 (+ 130us) server_negotiation.cc:300] Negotiation successful
1025 20:08:46.221895 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20251025 20:08:46.224429 25450 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:46.223616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.18.82.65:38901)
1025 20:08:46.223728 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.223733 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.223783 (+ 50us) server_negotiation.cc:408] Connection header received
1025 20:08:46.223892 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.223896 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.223932 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.223997 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.224343 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41214: BlockingRecv error: recv got EOF from 127.0.0.1:41214 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20251025 20:08:46.224519 18761 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761422869575105-18761-0/krb5kdc/krb5cc)
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.225353 18761 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:46.226233 18761 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[24930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.65@KRBTEST.COM
I20251025 20:08:46.232772 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:46.226963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.18.82.65:38901)
1025 20:08:46.227086 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:46.227089 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:46.227102 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:46.227194 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:46.227197 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:46.227228 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:46.227290 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:46.227797 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.228180 (+ 383us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:46.228671 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:46.228773 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:46.230493 (+ 1720us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:46.230507 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:46.230509 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:46.230530 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:46.231699 (+ 1169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.232010 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.232012 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.232014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.232044 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:46.232205 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:46.232207 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:46.232208 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:46.232308 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:46.232390 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:46.232531 (+ 141us) server_negotiation.cc:300] Negotiation successful
1025 20:08:46.232609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 20:08:46.233680 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25042
I20251025 20:08:46.240434 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25178
I20251025 20:08:46.246001 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25314
I20251025 20:08:46.251345 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 24946
2025-10-25T20:08:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2903 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[25463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[25463](info): set up 2 sockets
Oct 25 20:08:46 dist-test-slave-d2pk krb5kdc[25463](info): commencing operation
krb5kdc: starting...
W20251025 20:08:48.110097 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.837s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:48.234100 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:45153
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:45153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:48.334415 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:48.334627 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:48.334702 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:48.337694 25479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:48.337759 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:48.337796 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:48.337845 25479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:48.337882 25479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:48.341831 25479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:45153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:48.342832 25479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:48.343652 25479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:48.348968 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:48.348959 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:48.348950 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:48.349231 25479 server_base.cc:1047] running on GCE node
I20251025 20:08:48.349702 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:48.350543 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:48.351687 25479 hybrid_clock.cc:648] HybridClock initialized: now 1761422928351669 us; error 38 us; skew 500 ppm
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:48.354367 25479 init.cc:377] Logged in from keytab as oryx/127.18.82.126@KRBTEST.COM (short username oryx)
I20251025 20:08:48.355367 25479 webserver.cc:492] Webserver started at http://127.18.82.126:43007/ using document root <none> and password file <none>
I20251025 20:08:48.355888 25479 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:48.355938 25479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:48.356115 25479 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:48.357683 25479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "26cfb2914af2435b90f27010e9d4d8e6"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "10be06f608f781b56831bd35a70c1614"
server_key_iv: "7ce17dbca428876aa73377a43706fdf0"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.358111 25479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "26cfb2914af2435b90f27010e9d4d8e6"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "10be06f608f781b56831bd35a70c1614"
server_key_iv: "7ce17dbca428876aa73377a43706fdf0"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.361225 25479 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 20:08:48.368973 25494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.370193 25479 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20251025 20:08:48.370398 25479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "26cfb2914af2435b90f27010e9d4d8e6"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "10be06f608f781b56831bd35a70c1614"
server_key_iv: "7ce17dbca428876aa73377a43706fdf0"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.370489 25479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:48.387976 25479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:48.390722 25479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:48.390864 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:48.397161 25479 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:45153
I20251025 20:08:48.397183 25546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:45153 every 8 connection(s)
I20251025 20:08:48.398140 25479 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:48.398710 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25479
I20251025 20:08:48.398849 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:48.399060 18761 external_mini_cluster.cc:1468] Setting key 3a942cdc22ddab9f421b971f8d263c3e
I20251025 20:08:48.401034 25547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:48.407176 25547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Bootstrap starting.
I20251025 20:08:48.409756 25547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:48.410544 25547 log.cc:826] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:48.411896 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:48.400124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.18.82.126:45153)
1025 20:08:48.400496 (+ 372us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:48.400506 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:48.400541 (+ 35us) server_negotiation.cc:408] Connection header received
1025 20:08:48.401305 (+ 764us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:48.401328 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:48.401680 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:48.402022 (+ 342us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:48.402787 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.404009 (+ 1222us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:48.404668 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.404904 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:48.407213 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:48.407260 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:48.407273 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:48.407311 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:48.409210 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.409670 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.409678 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.409683 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.409771 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.410045 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.410048 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.410050 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.410415 (+ 365us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:48.410608 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:48.410958 (+ 350us) server_negotiation.cc:300] Negotiation successful
1025 20:08:48.411193 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20251025 20:08:48.412390 25547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: No bootstrap required, opened a new log
I20251025 20:08:48.415007 25547 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } }
I20251025 20:08:48.415226 25547 raft_consensus.cc:385] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:48.415297 25547 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cfb2914af2435b90f27010e9d4d8e6, State: Initialized, Role: FOLLOWER
I20251025 20:08:48.415704 25547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } }
I20251025 20:08:48.415853 25547 raft_consensus.cc:399] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:48.415920 25547 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:48.416010 25547 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:48.416777 25547 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } }
I20251025 20:08:48.417068 25547 leader_election.cc:304] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26cfb2914af2435b90f27010e9d4d8e6; no voters:
I20251025 20:08:48.417317 25547 leader_election.cc:290] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:48.417410 25552 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:48.417649 25552 raft_consensus.cc:697] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 LEADER]: Becoming Leader. State: Replica: 26cfb2914af2435b90f27010e9d4d8e6, State: Running, Role: LEADER
I20251025 20:08:48.417956 25552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } }
I20251025 20:08:48.418376 25547 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:48.419711 25554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cfb2914af2435b90f27010e9d4d8e6. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } }
I20251025 20:08:48.419842 25554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:48.420498 25553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } }
I20251025 20:08:48.420616 25553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
W20251025 20:08:48.423167 25568 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:48.423229 25568 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:48.423305 25561 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:48.423959 25561 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:48.428579 25561 catalog_manager.cc:1357] Generated new cluster ID: 8d84a661b37e46ec9999f3adb8a9e7cb
I20251025 20:08:48.428639 25561 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:48.443650 25561 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:48.444455 25561 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:48.449906 25561 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Generated new TSK 0
I20251025 20:08:48.450492 25561 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:48.497488 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:45153
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:48.599591 25575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:48.599800 25575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:48.599854 25575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:48.602938 25575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:48.602991 25575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:48.603102 25575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:48.607208 25575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:48.608238 25575 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:48.609030 25575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:48.615244 25580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:48.615353 25575 server_base.cc:1047] running on GCE node
W20251025 20:08:48.615275 25583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:48.615244 25581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:48.615911 25575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:48.616367 25575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:48.617547 25575 hybrid_clock.cc:648] HybridClock initialized: now 1761422928617529 us; error 26 us; skew 500 ppm
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:48.620065 25575 init.cc:377] Logged in from keytab as oryx/127.18.82.65@KRBTEST.COM (short username oryx)
I20251025 20:08:48.621068 25575 webserver.cc:492] Webserver started at http://127.18.82.65:35983/ using document root <none> and password file <none>
I20251025 20:08:48.621651 25575 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:48.621713 25575 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:48.621906 25575 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:48.623795 25575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "9719aa0b9f60432783f766daf87752fd"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "1bf29b0ce2e90231053cdf43b24e00e5"
server_key_iv: "ecf84b95fa027fdd843db95cd232143d"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.624243 25575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "9719aa0b9f60432783f766daf87752fd"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "1bf29b0ce2e90231053cdf43b24e00e5"
server_key_iv: "ecf84b95fa027fdd843db95cd232143d"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.627357 25575 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:48.629269 25590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.630223 25575 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:48.630323 25575 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "9719aa0b9f60432783f766daf87752fd"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "1bf29b0ce2e90231053cdf43b24e00e5"
server_key_iv: "ecf84b95fa027fdd843db95cd232143d"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.630401 25575 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:48.661113 25575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:48.663642 25575 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:48.663805 25575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:48.664304 25575 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:48.665112 25575 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:48.665158 25575 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.665210 25575 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:48.665241 25575 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.674062 25575 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:36767
I20251025 20:08:48.674116 25703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:36767 every 8 connection(s)
I20251025 20:08:48.674955 25575 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:48.682405 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25575
I20251025 20:08:48.682569 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:48.682859 18761 external_mini_cluster.cc:1468] Setting key 31d8b126c8c3281b2f16f56998642acf
I20251025 20:08:48.686213 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:48.676486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:39497 (local address 127.18.82.126:45153)
1025 20:08:48.676640 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:48.676645 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:48.677230 (+ 585us) server_negotiation.cc:408] Connection header received
1025 20:08:48.677980 (+ 750us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:48.677983 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:48.678031 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:48.678099 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:48.679175 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.679629 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:48.680142 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.680249 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:48.682781 (+ 2532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:48.682800 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:48.682803 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:48.682825 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:48.684358 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.684858 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.684860 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.684861 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.684893 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.685150 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.685152 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.685153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.685290 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:48.685352 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:48.685900 (+ 548us) server_negotiation.cc:300] Negotiation successful
1025 20:08:48.686036 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 20:08:48.686853 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:48.676730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:39497)
1025 20:08:48.677107 (+ 377us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:48.677138 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:48.677811 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:48.678226 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:48.678233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:48.678581 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:48.679060 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:48.679071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.679705 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:48.679707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:48.680063 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:48.680068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.680218 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:48.681193 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:48.681213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:48.682639 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:48.684461 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:48.684466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:48.684476 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:48.684778 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:48.684960 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:48.684966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:48.684970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:48.685077 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:48.685375 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:48.685382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:48.685658 (+ 276us) client_negotiation.cc:769] Sending connection context
1025 20:08:48.685863 (+ 205us) client_negotiation.cc:240] Negotiation successful
1025 20:08:48.686101 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20251025 20:08:48.688051 25704 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:48.688246 25704 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:48.688694 25704 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:48.690177 25511 ts_manager.cc:194] Registered new tserver with Master: 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
I20251025 20:08:48.691628 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:39497
WARNING: no policy specified for oryx/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:48.728581 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:45153
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:48.830734 25711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:48.830945 25711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:48.831028 25711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:48.834180 25711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:48.834239 25711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:48.834344 25711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:48.838356 25711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:48.839305 25711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:48.840128 25711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:48.846127 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:48.846160 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:48.846367 25711 server_base.cc:1047] running on GCE node
W20251025 20:08:48.846131 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:48.846715 25711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:48.847201 25711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:48.848349 25711 hybrid_clock.cc:648] HybridClock initialized: now 1761422928848325 us; error 36 us; skew 500 ppm
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:48.850800 25711 init.cc:377] Logged in from keytab as oryx/127.18.82.66@KRBTEST.COM (short username oryx)
I20251025 20:08:48.851684 25711 webserver.cc:492] Webserver started at http://127.18.82.66:38477/ using document root <none> and password file <none>
I20251025 20:08:48.852159 25711 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:48.852207 25711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:48.852370 25711 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:48.854000 25711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "9b446012218f4316a16864099e5bdbaf"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "2e1cb1ad0610db0a82a9a331690fa8e3"
server_key_iv: "7aae2bf4b1d62571487deae10b491f6c"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.854425 25711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "9b446012218f4316a16864099e5bdbaf"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "2e1cb1ad0610db0a82a9a331690fa8e3"
server_key_iv: "7aae2bf4b1d62571487deae10b491f6c"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.857390 25711 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:08:48.859387 25726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.860311 25711 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:48.860409 25711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "9b446012218f4316a16864099e5bdbaf"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "2e1cb1ad0610db0a82a9a331690fa8e3"
server_key_iv: "7aae2bf4b1d62571487deae10b491f6c"
server_key_version: "encryptionkey@0"
I20251025 20:08:48.860487 25711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:48.880939 25711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:48.883373 25711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:48.883535 25711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:48.884048 25711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:48.884832 25711 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:48.884881 25711 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.884934 25711 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:48.884967 25711 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:48.893267 25711 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:42231
I20251025 20:08:48.893312 25839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:42231 every 8 connection(s)
I20251025 20:08:48.894167 25711 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
Oct 25 20:08:48 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:48.903658 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25711
I20251025 20:08:48.903784 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:48.904073 18761 external_mini_cluster.cc:1468] Setting key 04369b872c3af120a883891b432582c9
I20251025 20:08:48.905849 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:48.895846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39861 (local address 127.18.82.126:45153)
1025 20:08:48.895986 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:48.895990 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:48.896603 (+ 613us) server_negotiation.cc:408] Connection header received
1025 20:08:48.897382 (+ 779us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:48.897386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:48.897445 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:48.897551 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:48.898726 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.899207 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:48.899805 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.900002 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:48.902476 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:48.902494 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:48.902496 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:48.902520 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:48.904070 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.904557 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.904560 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.904561 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.904610 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:48.904880 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:48.904883 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:48.904884 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:48.905022 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:48.905109 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:48.905644 (+ 535us) server_negotiation.cc:300] Negotiation successful
1025 20:08:48.905740 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:48.906774 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:48.896073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:39861)
1025 20:08:48.896477 (+ 404us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:48.896509 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:48.897196 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:48.897692 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:48.897701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:48.898045 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:48.898575 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:48.898587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.899319 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:48.899322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:48.899693 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:48.899700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:48.900359 (+ 659us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:48.900939 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:48.900966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:48.902291 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:48.904193 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:48.904198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:48.904206 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:48.904441 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:48.904707 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:48.904710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:48.904711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:48.904790 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:48.905123 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:48.905132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:48.905362 (+ 230us) client_negotiation.cc:769] Sending connection context
1025 20:08:48.905610 (+ 248us) client_negotiation.cc:240] Negotiation successful
1025 20:08:48.905872 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20251025 20:08:48.907811 25840 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:48.908046 25840 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:48.908622 25840 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:48.909737 25511 ts_manager.cc:194] Registered new tserver with Master: 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:48.910254 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:39861
WARNING: no policy specified for oryx/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:48.948390 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:45153
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:49.047021 25847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:49.047217 25847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:49.047291 25847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:49.050184 25847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:49.050230 25847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:49.050346 25847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:49.054303 25847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:49.055271 25847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:49.056074 25847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:49.062172 25853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.062184 25855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.062172 25852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.062470 25847 server_base.cc:1047] running on GCE node
I20251025 20:08:49.062853 25847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:49.063337 25847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:49.064479 25847 hybrid_clock.cc:648] HybridClock initialized: now 1761422929064468 us; error 25 us; skew 500 ppm
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:49.067193 25847 init.cc:377] Logged in from keytab as oryx/127.18.82.67@KRBTEST.COM (short username oryx)
I20251025 20:08:49.068182 25847 webserver.cc:492] Webserver started at http://127.18.82.67:43885/ using document root <none> and password file <none>
I20251025 20:08:49.068706 25847 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.068755 25847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.068933 25847 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:49.070549 25847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "14185135b8d443c2aeecd6d5717b930f"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "9ae9e322861348ddc4efbc74bdeade8c"
server_key_iv: "9986af2d5d3c85223c17540b76921431"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.071003 25847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "14185135b8d443c2aeecd6d5717b930f"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "9ae9e322861348ddc4efbc74bdeade8c"
server_key_iv: "9986af2d5d3c85223c17540b76921431"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.074290 25847 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:08:49.076435 25862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.077389 25847 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:49.077493 25847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "14185135b8d443c2aeecd6d5717b930f"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "9ae9e322861348ddc4efbc74bdeade8c"
server_key_iv: "9986af2d5d3c85223c17540b76921431"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.077621 25847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.105440 25847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:49.108009 25847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:49.108178 25847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:49.108659 25847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:49.109431 25847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:49.109477 25847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.109555 25847 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:49.109579 25847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.117619 25847 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:38011
I20251025 20:08:49.117641 25975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:38011 every 8 connection(s)
I20251025 20:08:49.118498 25847 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:49.123745 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25847
I20251025 20:08:49.123874 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:49.124099 18761 external_mini_cluster.cc:1468] Setting key b0c3c908ac3962f7eec5965e97c0f4a6
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:49.130393 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.120197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:57097 (local address 127.18.82.126:45153)
1025 20:08:49.120352 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.120355 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.121033 (+ 678us) server_negotiation.cc:408] Connection header received
1025 20:08:49.121830 (+ 797us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.121834 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.121910 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.121986 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.123161 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.123756 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.124435 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.124587 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.127206 (+ 2619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.127220 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.127222 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.127246 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.128634 (+ 1388us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.129063 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.129066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.129067 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.129109 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.129377 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.129380 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.129381 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.129549 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.129647 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.130109 (+ 462us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.130236 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:49.130959 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.120488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:57097)
1025 20:08:49.120897 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.120930 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.121647 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.122146 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.122153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.122505 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.123017 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.123028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.123890 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.123896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.124319 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.124330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.124540 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.125635 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.125671 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.127077 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.128722 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.128725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.128732 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.128955 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.129190 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.129194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.129196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.129273 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.129663 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.129672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.129880 (+ 208us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.130053 (+ 173us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.130290 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20251025 20:08:49.131888 25976 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:49.132081 25976 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:49.132515 25976 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:49.133405 25511 ts_manager.cc:194] Registered new tserver with Master: 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011)
I20251025 20:08:49.133983 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:57097
I20251025 20:08:49.137495 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.82.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:49.179960 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.125:45489
--webserver_interface=127.18.82.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:45153,127.18.82.125:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:49.278996 25983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:49.279188 25983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:49.279253 25983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:49.282315 25983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:49.282368 25983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:49.282392 25983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:49.282430 25983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:49.282469 25983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:49.286422 25983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:45153,127.18.82.125:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.125:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.82.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
W20251025 20:08:49.286585 25983 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.82.126:45153,127.18.82.125:45489'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251025 20:08:49.287355 25983 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:49.288126 25983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:49.293294 25989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.293314 25988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.293322 25991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.293438 25983 server_base.cc:1047] running on GCE node
I20251025 20:08:49.294059 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:49.294862 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:49.295996 25983 hybrid_clock.cc:648] HybridClock initialized: now 1761422929295981 us; error 30 us; skew 500 ppm
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:49.298496 25983 init.cc:377] Logged in from keytab as oryx/127.18.82.125@KRBTEST.COM (short username oryx)
I20251025 20:08:49.299454 25983 webserver.cc:492] Webserver started at http://127.18.82.125:34721/ using document root <none> and password file <none>
I20251025 20:08:49.299954 25983 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.300001 25983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.300177 25983 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:49.301771 25983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/instance:
uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "b1473b8fe6cb5819b7dcd0b2a306749d"
server_key_iv: "092a1d7a618b9af2cecfefe908bae89b"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.302232 25983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal/instance:
uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "b1473b8fe6cb5819b7dcd0b2a306749d"
server_key_iv: "092a1d7a618b9af2cecfefe908bae89b"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.305311 25983 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 20:08:49.307350 25998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.308281 25983 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251025 20:08:49.308377 25983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "b1473b8fe6cb5819b7dcd0b2a306749d"
server_key_iv: "092a1d7a618b9af2cecfefe908bae89b"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.308462 25983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.318786 25983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:49.321462 25983 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:49.321668 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:49.328253 25983 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.125:45489
I20251025 20:08:49.328264 26050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.125:45489 every 8 connection(s)
I20251025 20:08:49.329180 25983 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/info.pb
I20251025 20:08:49.331741 26051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:49.334555 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 25983
I20251025 20:08:49.335664 26051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.125@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
I20251025 20:08:49.346102 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.337183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.18.82.126:45153)
1025 20:08:49.337312 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.337317 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.337627 (+ 310us) server_negotiation.cc:408] Connection header received
1025 20:08:49.337870 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.337873 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.337927 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.338014 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.338924 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.339662 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.340419 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.340618 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.342727 (+ 2109us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.342747 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.342751 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.342780 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.344344 (+ 1564us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.344840 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.344845 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.344847 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.344887 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.345117 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.345120 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.345122 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.345276 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.345360 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.345821 (+ 461us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.345931 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:49.346755 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.337103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33868)
1025 20:08:49.337472 (+ 369us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.337552 (+ 80us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.337741 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.338080 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.338088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.338546 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.338793 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.338803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.339752 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.339756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.340308 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.340317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.340523 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.340993 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.341027 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.342587 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.344446 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.344453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.344460 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.344752 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.344963 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.344966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.344967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.345038 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.345380 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.345390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.345618 (+ 228us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.345773 (+ 155us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.346020 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
I20251025 20:08:49.346760 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.335905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.18.82.125:45489)
1025 20:08:49.336278 (+ 373us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.336286 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.336316 (+ 30us) server_negotiation.cc:408] Connection header received
1025 20:08:49.336847 (+ 531us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.336870 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.337212 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.337574 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.338568 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.339517 (+ 949us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.340314 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.340961 (+ 647us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.343175 (+ 2214us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.343191 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.343198 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.343219 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.344810 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.345149 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.345153 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.345154 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.345196 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.345390 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.345393 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.345394 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.345626 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.345784 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.345994 (+ 210us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.346236 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"spinlock_wait_cycles":19712,"thread_start_us":117,"threads_started":1}
I20251025 20:08:49.348101 26051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.125@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
I20251025 20:08:49.354351 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.348634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.18.82.125:45489)
1025 20:08:49.348716 (+ 82us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.348721 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.348812 (+ 91us) server_negotiation.cc:408] Connection header received
1025 20:08:49.348898 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.348901 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.348933 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.348989 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.349418 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.349880 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.350349 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.350517 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.351820 (+ 1303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.351831 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.351833 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.351851 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.353090 (+ 1239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.353431 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.353437 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.353439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.353473 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.353746 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.353749 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.353751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.353869 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.353943 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.354127 (+ 184us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.354235 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":35}
I20251025 20:08:49.354355 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.348557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42700)
1025 20:08:49.348716 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.348730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.348803 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.349019 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.349021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.349177 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.349317 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.349322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.349975 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.349978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.350266 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.350271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.350366 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.350756 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.350765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.351719 (+ 954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.353186 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.353189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.353191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.353340 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.353588 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.353591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.353593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.353651 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.353944 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.353947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.354008 (+ 61us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.354083 (+ 75us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.354189 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20251025 20:08:49.356700 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Bootstrap starting.
I20251025 20:08:49.358952 26051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:49.359620 26051 log.cc:826] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:49.361109 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: No bootstrap required, opened a new log
I20251025 20:08:49.363451 26051 raft_consensus.cc:359] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } }
I20251025 20:08:49.363607 26051 raft_consensus.cc:385] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:49.363653 26051 raft_consensus.cc:740] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e87b1cb0ca4b91aec6bc3ff3426fe4, State: Initialized, Role: FOLLOWER
I20251025 20:08:49.364013 26051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } }
I20251025 20:08:49.364389 26057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } } }
I20251025 20:08:49.364486 26057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:49.364975 26051 sys_catalog.cc:565] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: configured and running, proceeding with master startup.
W20251025 20:08:49.369127 26068 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:49.369201 26068 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:49.370688 25983 master_runner.cc:428] Detected that this master 83e87b1cb0ca4b91aec6bc3ff3426fe4 is joining an existing cluster
I20251025 20:08:49.370733 25983 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.82.125:45489
I20251025 20:08:49.379882 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.372717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.18.82.126:45153)
1025 20:08:49.372843 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.372847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.373169 (+ 322us) server_negotiation.cc:408] Connection header received
1025 20:08:49.373315 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.373317 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.373381 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.373459 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.374745 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.375425 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.376250 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.376756 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.377187 (+ 431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.377199 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.377201 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.377216 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.378612 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.379025 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.379027 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.379029 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.379061 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.379255 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.379257 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.379258 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.379382 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.379447 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.379704 (+ 257us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.379779 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 20:08:49.380038 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.372446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33884)
1025 20:08:49.373092 (+ 646us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.373104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.373198 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.373483 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.373486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.373835 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.374579 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.374596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.375576 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.375580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.376127 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.376138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.376261 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.376741 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.376756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.377086 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.378720 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.378725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.378727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.378939 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.379128 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.379132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.379134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.379185 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.379459 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.379463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.379563 (+ 100us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.379675 (+ 112us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.379788 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":224,"threads_started":1}
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.126@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
I20251025 20:08:49.396908 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.372850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42708)
1025 20:08:49.373175 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.373189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.373300 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.373626 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.373628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.373826 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.374041 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.374046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.374700 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.374702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.375144 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.375152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.375523 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.375994 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.376006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.376272 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.395480 (+ 19208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.395483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.395485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.395795 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.395998 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.396005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.396044 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.396339 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.396343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.396414 (+ 71us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.396524 (+ 110us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.396684 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20251025 20:08:49.396916 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.372562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.18.82.125:45489)
1025 20:08:49.372683 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.372690 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.373257 (+ 567us) server_negotiation.cc:408] Connection header received
1025 20:08:49.373395 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.373398 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.373444 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.373537 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.374138 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.374596 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.375247 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.375410 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.376393 (+ 983us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.376410 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.376414 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.376440 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.395373 (+ 18933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.395886 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.395890 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.395891 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.395927 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396120 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.396123 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.396124 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.396251 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.396336 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.396577 (+ 241us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.396748 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":34}
I20251025 20:08:49.398764 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.383643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42720)
1025 20:08:49.383993 (+ 350us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.384021 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.384144 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.384415 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.384423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.384747 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.384917 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.384922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.385920 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.385924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.386240 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.386247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.386398 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.386932 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.386944 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.388379 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.396073 (+ 7694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396083 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.396089 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.396367 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.396601 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.396607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.396668 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.398200 (+ 1532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.398206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.398321 (+ 115us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.398464 (+ 143us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.398618 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
I20251025 20:08:49.398774 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.383668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.18.82.125:45489)
1025 20:08:49.383924 (+ 256us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.383928 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.384084 (+ 156us) server_negotiation.cc:408] Connection header received
1025 20:08:49.384256 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.384259 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.384297 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.384369 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.385050 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.385814 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.386390 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.386614 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.388497 (+ 1883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.388510 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.388512 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.388531 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.395945 (+ 7414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396453 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.396456 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.396457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.396496 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.396773 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.396778 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.396780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.396922 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.398152 (+ 1230us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.398524 (+ 372us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.398626 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"spinlock_wait_cycles":3328,"thread_start_us":111,"threads_started":1}
I20251025 20:08:49.402354 25511 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.18.82.125:45489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
member_type: NON_VOTER
last_known_addr {
host: "127.18.82.125"
port: 45489
}
attrs {
promote: true
}
}
dest_uuid: "26cfb2914af2435b90f27010e9d4d8e6"
cas_config_opid_index: -1
I20251025 20:08:49.402915 25511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } }
I20251025 20:08:49.405123 26025 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:49.406481 26025 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251025 20:08:49.407262 26080 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.18.82.124@KRBTEST.COM; defaulting to no policy
I20251025 20:08:49.408383 26081 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:49.409947 26080 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.18.82.125:45489
I20251025 20:08:49.410537 25983 master.cc:561] Master@127.18.82.125:45489 shutting down...
I20251025 20:08:49.410660 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.124@KRBTEST.COM" created.
I20251025 20:08:49.413993 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:49.416122 26025 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
I20251025 20:08:49.418212 26057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cfb2914af2435b90f27010e9d4d8e6. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
I20251025 20:08:49.418365 26057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.418231 26086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
I20251025 20:08:49.418473 26086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.417822 26087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
I20251025 20:08:49.418702 26087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.420058 26082 mvcc.cc:204] Tried to move back new op lower bound from 7214788314927353856 to 7214788314800701440. Current Snapshot: MvccSnapshot[applied={T|T < 7214788314834845696}]
I20251025 20:08:49.423002 25983 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Raft consensus shutting down.
I20251025 20:08:49.423295 25983 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Raft consensus is shut down!
I20251025 20:08:49.423398 25983 tablet_replica.cc:333] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.124@KRBTEST.COM; defaulting to no policy
I20251025 20:08:49.438212 25983 master.cc:583] Master@127.18.82.125:45489 shutdown complete.
I20251025 20:08:49.438256 25983 master_runner.cc:305] Clearing existing system tablet
I20251025 20:08:49.438503 25983 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.438560 25983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.124@KRBTEST.COM" created.
I20251025 20:08:49.439666 25983 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:49.440579 26094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.440763 25983 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.440866 25983 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:49.440900 25983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "b1473b8fe6cb5819b7dcd0b2a306749d"
server_key_iv: "092a1d7a618b9af2cecfefe908bae89b"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.442620 25983 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:49.444737 25983 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251025 20:08:49.444828 25983 log.cc:1199] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251025 20:08:49.445122 25983 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Deleting consensus metadata
I20251025 20:08:49.445279 25983 master_runner.cc:315] Copying system tablet from 127.18.82.126:45153
I20251025 20:08:49.446460 25983 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: tablet copy: Beginning tablet copy session from remote peer at address 127.18.82.126:45153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:49.452836 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.124:44491
--webserver_interface=127.18.82.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 20:08:49.454684 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.447138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.18.82.126:45153)
1025 20:08:49.447263 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.447275 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.447336 (+ 61us) server_negotiation.cc:408] Connection header received
1025 20:08:49.447542 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.447545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.447592 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.447658 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.448246 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.448759 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.449539 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.449674 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.450587 (+ 913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.450603 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.450606 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.450638 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.453015 (+ 2377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.453529 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.453536 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.453541 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.453613 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.453923 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.453929 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.453933 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.454118 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.454184 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.454466 (+ 282us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.454557 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:49.454702 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.446977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33888)
1025 20:08:49.447245 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.447260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.447415 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.447691 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.447695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.447878 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.448124 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.448131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.448866 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.448869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.449409 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.449418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.449569 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.450103 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.450118 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.450462 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.453149 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.453155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.453160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.453393 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.453744 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.453749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.453753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.453831 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.454238 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.454240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.454307 (+ 67us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.454399 (+ 92us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.454524 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20251025 20:08:49.455224 25531 tablet_copy_service.cc:140] P 26cfb2914af2435b90f27010e9d4d8e6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 83e87b1cb0ca4b91aec6bc3ff3426fe4 ({username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:33888)
I20251025 20:08:49.455364 25531 tablet_copy_service.cc:161] P 26cfb2914af2435b90f27010e9d4d8e6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 83e87b1cb0ca4b91aec6bc3ff3426fe4 at {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:33888: session id = 83e87b1cb0ca4b91aec6bc3ff3426fe4-00000000000000000000000000000000
I20251025 20:08:49.456691 25531 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Tablet Copy: opened 0 blocks and 1 log segments
I20251025 20:08:49.457994 25983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251025 20:08:49.459460 25983 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: tablet copy: Starting download of 0 data blocks...
I20251025 20:08:49.459635 25983 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: tablet copy: Starting download of 1 WAL segments...
I20251025 20:08:49.461200 25983 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251025 20:08:49.462723 25531 tablet_copy_service.cc:342] P 26cfb2914af2435b90f27010e9d4d8e6: Request end of tablet copy session 83e87b1cb0ca4b91aec6bc3ff3426fe4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:33888
I20251025 20:08:49.462824 25531 tablet_copy_service.cc:434] P 26cfb2914af2435b90f27010e9d4d8e6: ending tablet copy session 83e87b1cb0ca4b91aec6bc3ff3426fe4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 83e87b1cb0ca4b91aec6bc3ff3426fe4
W20251025 20:08:49.466728 25983 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251025 20:08:49.468685 25983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:49.470124 26102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.470238 26103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.474368 25983 server_base.cc:1047] running on GCE node
W20251025 20:08:49.474313 26105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.474649 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:49.474902 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:49.476039 25983 hybrid_clock.cc:648] HybridClock initialized: now 1761422929476032 us; error 22 us; skew 500 ppm
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:49.477536 25983 init.cc:377] Logged in from keytab as oryx/127.18.82.125@KRBTEST.COM (short username oryx)
I20251025 20:08:49.477957 25983 webserver.cc:492] Webserver started at http://127.18.82.125:35745/ using document root <none> and password file <none>
I20251025 20:08:49.478125 25983 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.478163 25983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.479136 25983 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:49.479907 26112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.480136 25983 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:49.480201 25983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "b1473b8fe6cb5819b7dcd0b2a306749d"
server_key_iv: "092a1d7a618b9af2cecfefe908bae89b"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.480276 25983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.484200 25983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:49.485952 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:49.490229 25983 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.125:45489
I20251025 20:08:49.490249 26164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.125:45489 every 8 connection(s)
I20251025 20:08:49.490707 25983 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-1/data/info.pb
I20251025 20:08:49.491983 26165 sys_catalog.cc:263] Verifying existing consensus state
I20251025 20:08:49.492890 26165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Bootstrap starting.
I20251025 20:08:49.498526 26165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 20:08:49.498827 26165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Bootstrap complete.
I20251025 20:08:49.499300 26165 raft_consensus.cc:359] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } }
I20251025 20:08:49.499410 26165 raft_consensus.cc:740] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e87b1cb0ca4b91aec6bc3ff3426fe4, State: Initialized, Role: LEARNER
I20251025 20:08:49.499527 26165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } }
I20251025 20:08:49.499861 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } }
I20251025 20:08:49.500015 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.500041 26165 sys_catalog.cc:565] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:49.502853 26179 catalog_manager.cc:1269] Loaded cluster ID: 8d84a661b37e46ec9999f3adb8a9e7cb
I20251025 20:08:49.502947 26179 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: loading cluster ID for follower catalog manager: success
I20251025 20:08:49.504563 26179 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: acquiring CA information for follower catalog manager: success
I20251025 20:08:49.505585 26179 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251025 20:08:49.596438 26101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:49.596695 26101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:49.596766 26101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:49.600436 26101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:49.600494 26101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:49.600515 26101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:49.600531 26101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:49.600548 26101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:49.604434 26101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.124:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.82.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:49.605394 26101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:49.606226 26101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:49.611332 26184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.611402 26187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.611387 26185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.611697 26101 server_base.cc:1047] running on GCE node
I20251025 20:08:49.612171 26101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:49.613036 26101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:49.614178 26101 hybrid_clock.cc:648] HybridClock initialized: now 1761422929614163 us; error 29 us; skew 500 ppm
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:49.616677 26101 init.cc:377] Logged in from keytab as oryx/127.18.82.124@KRBTEST.COM (short username oryx)
I20251025 20:08:49.617647 26101 webserver.cc:492] Webserver started at http://127.18.82.124:41795/ using document root <none> and password file <none>
I20251025 20:08:49.618167 26101 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.618220 26101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.618393 26101 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:49.620059 26101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/instance:
uuid: "0ec99364029d468c9bc52518dce6b3ad"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "35bb250ed2319db3b69ab63fd773f11b"
server_key_iv: "e818741aeae3b1b933abc6f3b5f628a6"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.620482 26101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal/instance:
uuid: "0ec99364029d468c9bc52518dce6b3ad"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "35bb250ed2319db3b69ab63fd773f11b"
server_key_iv: "e818741aeae3b1b933abc6f3b5f628a6"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.623597 26101 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 20:08:49.625485 26194 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.626412 26101 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:49.626513 26101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
uuid: "0ec99364029d468c9bc52518dce6b3ad"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "35bb250ed2319db3b69ab63fd773f11b"
server_key_iv: "e818741aeae3b1b933abc6f3b5f628a6"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.626606 26101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.636176 26101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:49.638494 26101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:49.638652 26101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:49.645057 26246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.124:44491 every 8 connection(s)
I20251025 20:08:49.645063 26101 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.124:44491
I20251025 20:08:49.645977 26101 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/info.pb
I20251025 20:08:49.648483 26247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:49.649206 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 26101
I20251025 20:08:49.652513 26247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.124@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:49.660926 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.650374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.18.82.124:44491)
1025 20:08:49.650698 (+ 324us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.650706 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.650735 (+ 29us) server_negotiation.cc:408] Connection header received
1025 20:08:49.651215 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.651248 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.651501 (+ 253us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.651745 (+ 244us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.652749 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.653554 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.654419 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.654745 (+ 326us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.656673 (+ 1928us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.656710 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.656718 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.656748 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.658688 (+ 1940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.659098 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.659103 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.659106 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.659159 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.659396 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.659400 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.659402 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.659710 (+ 308us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.659872 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.660145 (+ 273us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.660379 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20251025 20:08:49.661543 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.654156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.18.82.126:45153)
1025 20:08:49.654334 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.654341 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.654417 (+ 76us) server_negotiation.cc:408] Connection header received
1025 20:08:49.654625 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.654629 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.654683 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.654752 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.655560 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.656061 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.656714 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.656915 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.658733 (+ 1818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.658756 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.658760 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.658787 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.660118 (+ 1331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.660535 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.660538 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.660539 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.660589 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.660803 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.660806 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.660808 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.660932 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.661014 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.661328 (+ 314us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.661416 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:49.661576 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.653977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33894)
1025 20:08:49.654304 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.654340 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.654499 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.654812 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.654818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.655145 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.655426 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.655432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.656170 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.656176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.656591 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.656600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.656713 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.657303 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.657315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.658619 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.660227 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.660234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.660239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.660440 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.660670 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.660674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.660675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.660717 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.661024 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.661032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.661135 (+ 103us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.661266 (+ 131us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.661399 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":11520,"thread_start_us":114,"threads_started":1}
I20251025 20:08:49.662659 26247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.124@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
I20251025 20:08:49.669250 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.663052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42730)
1025 20:08:49.663361 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.663372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.663432 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.663646 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.663648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.663781 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.663967 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.663973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.664829 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.664832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.665179 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.665186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.665268 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.665706 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.665717 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.666750 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.668388 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.668390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.668391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.668511 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.668668 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.668670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.668671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.668701 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.668943 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.668945 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.669003 (+ 58us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.669076 (+ 73us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.669147 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20251025 20:08:49.669313 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.663169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.18.82.125:45489)
1025 20:08:49.663412 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.663416 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.663429 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:49.663505 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.663507 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.663548 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.663629 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.664052 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.664718 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.665267 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.665420 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.666865 (+ 1445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.666882 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.666884 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.666907 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.668304 (+ 1397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.668572 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.668574 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.668576 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.668615 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.668744 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.668746 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.668747 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.668854 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.668924 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.669104 (+ 180us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.669181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20251025 20:08:49.669948 26247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.124@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
I20251025 20:08:49.675928 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.670300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59494)
1025 20:08:49.670378 (+ 78us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.670388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.670444 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.670671 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.670673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.670774 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.670895 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.670899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.671448 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.671450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.671719 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.671724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.671796 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.672161 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.672172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.673220 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.675022 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.675025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.675026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.675156 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.675346 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.675347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.675349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.675378 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.675655 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.675657 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.675712 (+ 55us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.675775 (+ 63us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.675845 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20251025 20:08:49.676062 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.670412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.18.82.124:44491)
1025 20:08:49.670507 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.670510 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.670520 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:08:49.670545 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.670547 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.670577 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.670654 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.670968 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.671379 (+ 411us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.671787 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.671915 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.673333 (+ 1418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.673346 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.673349 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.673369 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.674931 (+ 1562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.675234 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.675236 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.675238 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.675278 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.675444 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.675446 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.675447 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.675575 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.675673 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.675839 (+ 166us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.675948 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251025 20:08:49.678277 26247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Bootstrap starting.
I20251025 20:08:49.680382 26247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:49.681052 26247 log.cc:826] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:49.682569 26247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: No bootstrap required, opened a new log
I20251025 20:08:49.684741 26247 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } }
I20251025 20:08:49.684880 26247 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:49.684940 26247 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec99364029d468c9bc52518dce6b3ad, State: Initialized, Role: FOLLOWER
I20251025 20:08:49.685308 26247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } }
I20251025 20:08:49.685696 26253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } } }
I20251025 20:08:49.685786 26253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:49.686224 26247 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: configured and running, proceeding with master startup.
W20251025 20:08:49.690502 26264 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 20:08:49.690568 26264 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 20:08:49.693081 26101 master_runner.cc:428] Detected that this master 0ec99364029d468c9bc52518dce6b3ad is joining an existing cluster
I20251025 20:08:49.693130 26101 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.82.124:44491
I20251025 20:08:49.694188 25704 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:49.722186 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.714830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33912)
1025 20:08:49.715045 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.715057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.715128 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.718990 (+ 3862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.718994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.719012 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:49.719246 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.719258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.721094 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.721097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.721751 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.721765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.721874 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.721936 (+ 62us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.721992 (+ 56us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.722052 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20251025 20:08:49.722632 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.715240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.18.82.126:45153)
1025 20:08:49.718729 (+ 3489us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.718734 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.718747 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:49.718804 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.718808 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.718853 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.719416 (+ 563us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:49.720141 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.720992 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.721859 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.722275 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.722341 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.722405 (+ 64us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.722471 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3396,"thread_start_us":103,"threads_started":1}
I20251025 20:08:49.724432 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.695536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.18.82.124:44491)
1025 20:08:49.695653 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.695656 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.695813 (+ 157us) server_negotiation.cc:408] Connection header received
1025 20:08:49.695931 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.695934 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.695969 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.696010 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.716050 (+ 20040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.716683 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.717396 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.717613 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.719011 (+ 1398us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.719026 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.719029 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.719050 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.721482 (+ 2432us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723482 (+ 2000us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.723486 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.723488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.723525 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723758 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.723761 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.723763 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.723910 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.723967 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.724203 (+ 236us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.724301 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251025 20:08:49.724870 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.695325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.18.82.126:45153)
1025 20:08:49.695463 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.695469 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.696063 (+ 594us) server_negotiation.cc:408] Connection header received
1025 20:08:49.696248 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.696252 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.696306 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.696371 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.714205 (+ 17834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.714932 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.715874 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.716057 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.719698 (+ 3641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.719716 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.719719 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.719744 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.722732 (+ 2988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723727 (+ 995us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.723731 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.723732 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.723781 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.724054 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.724060 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.724062 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.724217 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.724282 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.724635 (+ 353us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.724747 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:08:49.725224 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.695269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59500)
1025 20:08:49.695712 (+ 443us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.695726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.695836 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.696117 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.696120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.715630 (+ 19510us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.715927 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.715935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.716785 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.716788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.717277 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.717284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.717725 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.718442 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.718461 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.718883 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.723180 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.723186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.723388 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.723623 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.723628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.723670 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.724016 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.724019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.724074 (+ 55us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.724785 (+ 711us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.724891 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":83,"threads_started":1}
I20251025 20:08:49.725363 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.695747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33898)
1025 20:08:49.695978 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.695991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.696133 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.696486 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.696488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.713755 (+ 17267us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.714044 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.714054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.715094 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.715097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.715747 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.715755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.716169 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.719143 (+ 2974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.719157 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.719533 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.723379 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.723384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.723586 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.723899 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.723904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.723908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.723957 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.724340 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.724342 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.724418 (+ 76us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.725026 (+ 608us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.725163 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20251025 20:08:49.725682 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.713751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42742)
1025 20:08:49.714433 (+ 682us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.714446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.714565 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.714877 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.714880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.715625 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.715865 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.715871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.717313 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.717316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.717937 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.717949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.718407 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.719540 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.719554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.719943 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.724324 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.724328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.724331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.724540 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.724785 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.724788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.724790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.724832 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.725212 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.725215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.725289 (+ 74us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.725389 (+ 100us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.725502 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"spinlock_wait_cycles":3328,"thread_start_us":98,"threads_started":1}
I20251025 20:08:49.726130 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.714105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.18.82.125:45489)
1025 20:08:49.714237 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.714241 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.714504 (+ 263us) server_negotiation.cc:408] Connection header received
1025 20:08:49.714666 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.714670 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.714718 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.714763 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.715985 (+ 1222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.717180 (+ 1195us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.718071 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.718259 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.720134 (+ 1875us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.720154 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.720157 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.720186 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.723260 (+ 3074us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.724636 (+ 1376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.724640 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.724643 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.724684 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.724917 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.724920 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.724922 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.725089 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.725173 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.725888 (+ 715us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.725998 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:49.729282 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.723741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42744)
1025 20:08:49.724163 (+ 422us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.724175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.724235 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.727230 (+ 2995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.727236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.727257 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:49.727438 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.727460 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.728229 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.728232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.728958 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.728969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.729065 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.729080 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.729129 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.729183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358}
I20251025 20:08:49.729936 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.723968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.18.82.125:45489)
1025 20:08:49.726252 (+ 2284us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.726256 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.726267 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:49.726334 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.726339 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.726427 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.726514 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:49.727586 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.728116 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.729077 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.729557 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.729704 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.729768 (+ 64us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.729814 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2234,"thread_start_us":75,"threads_started":1}
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422928, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.126@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
I20251025 20:08:49.736377 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.729077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59514)
1025 20:08:49.729436 (+ 359us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.729449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.729542 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.729910 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.729913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.730082 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.730251 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.730259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.731134 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.731137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.731506 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.731515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.731602 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.732009 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.732021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.733329 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.735420 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.735423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.735424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.735580 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.735773 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.735775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.735776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.735810 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.736073 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.736075 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.736130 (+ 55us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.736195 (+ 65us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.736271 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":84,"threads_started":1}
I20251025 20:08:49.736464 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.729355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.18.82.124:44491)
1025 20:08:49.729633 (+ 278us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.729637 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.729650 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:49.729717 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.729720 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.729786 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.729867 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.730379 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.731034 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.731610 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.731786 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.733462 (+ 1676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.733480 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.733484 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.733533 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.735324 (+ 1791us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.735658 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.735661 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.735663 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.735703 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.735870 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.735873 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.735875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.736005 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.736088 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.736248 (+ 160us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.736340 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20251025 20:08:49.737215 25510 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.18.82.124:44491: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad"
member_type: NON_VOTER
last_known_addr {
host: "127.18.82.124"
port: 44491
}
attrs {
promote: true
}
}
dest_uuid: "26cfb2914af2435b90f27010e9d4d8e6"
cas_config_opid_index: 5
I20251025 20:08:49.737782 25510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } }
I20251025 20:08:49.738651 26080 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.739223 26221 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:49.739840 26080 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.18.82.124:44491
I20251025 20:08:49.740010 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.740074 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:49.740286 26139 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251025 20:08:49.740484 26221 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251025 20:08:49.740754 26101 master.cc:561] Master@127.18.82.124:44491 shutting down...
I20251025 20:08:49.740722 26081 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251025 20:08:49.741824 26139 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.743227 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cfb2914af2435b90f27010e9d4d8e6. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.743324 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.743257 26277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: NON_VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: true } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.743489 26277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.743642 26081 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 20:08:49.745707 25495 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20251025 20:08:49.746191 26081 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: attempting to promote NON_VOTER 83e87b1cb0ca4b91aec6bc3ff3426fe4 to VOTER
I20251025 20:08:49.746240 26101 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:49.746500 26101 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:49.746620 26101 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: stopping tablet replica
I20251025 20:08:49.746589 26081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } }
I20251025 20:08:49.747777 26139 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 LEARNER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251025 20:08:49.748123 26079 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251025 20:08:49.749122 25495 consensus_peers.cc:597] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 -> Peer 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124:44491): Couldn't send request to peer 0ec99364029d468c9bc52518dce6b3ad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251025 20:08:49.749879 26079 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.750998 26079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.751102 26079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:49.751222 26079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.751331 26079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:49.752228 26139 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.753324 26277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.753432 26277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:49.754814 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.754555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59522)
1025 20:08:49.754706 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.754758 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251025 20:08:49.754993 25511 master.cc:679] Network error: unable to get registration information for peer 0ec99364029d468c9bc52518dce6b3ad: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
I20251025 20:08:49.763175 26101 master.cc:583] Master@127.18.82.124:44491 shutdown complete.
I20251025 20:08:49.763227 26101 master_runner.cc:305] Clearing existing system tablet
I20251025 20:08:49.763442 26101 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.763495 26101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.764720 26101 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251025 20:08:49.765748 26280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.766088 26101 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.766208 26101 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:49.766247 26101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
uuid: "0ec99364029d468c9bc52518dce6b3ad"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "35bb250ed2319db3b69ab63fd773f11b"
server_key_iv: "e818741aeae3b1b933abc6f3b5f628a6"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.766716 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.766481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59536)
1025 20:08:49.766614 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.766661 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20251025 20:08:49.766974 25511 master.cc:679] Network error: unable to get registration information for peer 0ec99364029d468c9bc52518dce6b3ad: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
I20251025 20:08:49.768239 26101 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:49.770185 26101 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251025 20:08:49.770259 26101 log.cc:1199] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251025 20:08:49.770488 26101 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Deleting consensus metadata
I20251025 20:08:49.770641 26101 master_runner.cc:315] Copying system tablet from 127.18.82.126:45153
I20251025 20:08:49.771642 26101 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: tablet copy: Beginning tablet copy session from remote peer at address 127.18.82.126:45153
I20251025 20:08:49.778523 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.772103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.18.82.126:45153)
1025 20:08:49.772213 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.772217 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.772521 (+ 304us) server_negotiation.cc:408] Connection header received
1025 20:08:49.772671 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.772673 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.772712 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.772771 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:49.773421 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.773984 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.774789 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.774950 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.775835 (+ 885us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:49.775851 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:49.775853 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:49.775874 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:49.777193 (+ 1319us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.777623 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.777626 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.777627 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.777667 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:49.777900 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:49.777902 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:49.777903 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:49.778028 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:49.778094 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.778325 (+ 231us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.778419 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:08:49.778590 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.772013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33918)
1025 20:08:49.772447 (+ 434us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.772460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.772558 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.772803 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.772806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.773017 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:49.773302 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.773309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.774079 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.774082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.774656 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.774665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.774786 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.775341 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:49.775357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:49.775714 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:49.777292 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.777295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.777297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.777500 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.777765 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:49.777769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:49.777771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:49.777825 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:49.778102 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:49.778105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:49.778192 (+ 87us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.778295 (+ 103us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.778416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":87,"threads_started":1}
I20251025 20:08:49.778954 25531 tablet_copy_service.cc:140] P 26cfb2914af2435b90f27010e9d4d8e6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ec99364029d468c9bc52518dce6b3ad ({username='oryx', principal='oryx/127.18.82.124@KRBTEST.COM'} at 127.0.0.1:33918)
I20251025 20:08:49.779031 25531 tablet_copy_service.cc:161] P 26cfb2914af2435b90f27010e9d4d8e6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ec99364029d468c9bc52518dce6b3ad at {username='oryx', principal='oryx/127.18.82.124@KRBTEST.COM'} at 127.0.0.1:33918: session id = 0ec99364029d468c9bc52518dce6b3ad-00000000000000000000000000000000
I20251025 20:08:49.779793 25531 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Tablet Copy: opened 0 blocks and 1 log segments
I20251025 20:08:49.780891 26101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251025 20:08:49.782591 26101 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: tablet copy: Starting download of 0 data blocks...
I20251025 20:08:49.782735 26101 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: tablet copy: Starting download of 1 WAL segments...
I20251025 20:08:49.784199 26101 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251025 20:08:49.785698 25531 tablet_copy_service.cc:342] P 26cfb2914af2435b90f27010e9d4d8e6: Request end of tablet copy session 0ec99364029d468c9bc52518dce6b3ad-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.82.124@KRBTEST.COM'} at 127.0.0.1:33918
I20251025 20:08:49.785785 25531 tablet_copy_service.cc:434] P 26cfb2914af2435b90f27010e9d4d8e6: ending tablet copy session 0ec99364029d468c9bc52518dce6b3ad-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ec99364029d468c9bc52518dce6b3ad
I20251025 20:08:49.786295 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.786037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59544)
1025 20:08:49.786163 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.786214 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20251025 20:08:49.786504 25511 master.cc:679] Network error: unable to get registration information for peer 0ec99364029d468c9bc52518dce6b3ad: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
W20251025 20:08:49.790274 26101 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251025 20:08:49.792405 26101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:49.793987 26287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.794049 26288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:49.794078 26290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:49.794237 26101 server_base.cc:1047] running on GCE node
I20251025 20:08:49.794375 26101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:49.794616 26101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:49.795744 26101 hybrid_clock.cc:648] HybridClock initialized: now 1761422929795738 us; error 23 us; skew 500 ppm
Oct 25 20:08:49 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422929, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:49.797262 26101 init.cc:377] Logged in from keytab as oryx/127.18.82.124@KRBTEST.COM (short username oryx)
I20251025 20:08:49.797780 26101 webserver.cc:492] Webserver started at http://127.18.82.124:46473/ using document root <none> and password file <none>
I20251025 20:08:49.797958 26101 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:49.798000 26101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:49.799191 26101 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251025 20:08:49.800110 26297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:49.800351 26101 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251025 20:08:49.800410 26101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
uuid: "0ec99364029d468c9bc52518dce6b3ad"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "35bb250ed2319db3b69ab63fd773f11b"
server_key_iv: "e818741aeae3b1b933abc6f3b5f628a6"
server_key_version: "encryptionkey@0"
I20251025 20:08:49.800483 26101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:49.822185 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.821944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59560)
1025 20:08:49.822061 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.822111 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251025 20:08:49.822383 25511 master.cc:679] Network error: unable to get registration information for peer 0ec99364029d468c9bc52518dce6b3ad: Client connection negotiation failed: client connection to 127.18.82.124:44491: connect: Connection refused (error 111)
I20251025 20:08:49.837998 26101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:49.839607 26101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:49.844022 26101 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.124:44491
I20251025 20:08:49.844048 26349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.124:44491 every 8 connection(s)
I20251025 20:08:49.844537 26101 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/master-2/data/info.pb
I20251025 20:08:49.846243 26350 sys_catalog.cc:263] Verifying existing consensus state
I20251025 20:08:49.847272 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Bootstrap starting.
I20251025 20:08:49.857748 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 20:08:49.858000 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: Bootstrap complete.
I20251025 20:08:49.858651 26350 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } }
I20251025 20:08:49.858793 26350 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ec99364029d468c9bc52518dce6b3ad, State: Initialized, Role: LEARNER
I20251025 20:08:49.858899 26350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } }
I20251025 20:08:49.859207 26353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:49.859318 26353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:49.859349 26350 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:49.862699 26364 catalog_manager.cc:1269] Loaded cluster ID: 8d84a661b37e46ec9999f3adb8a9e7cb
I20251025 20:08:49.862744 26364 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: loading cluster ID for follower catalog manager: success
I20251025 20:08:49.864319 26364 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: acquiring CA information for follower catalog manager: success
I20251025 20:08:49.865370 26364 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 20:08:49.892275 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.889729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59566)
1025 20:08:49.889858 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:49.889869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:49.889940 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:49.890262 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:49.890265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:49.890283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:49.890451 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.890456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.891377 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.891380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:49.891915 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:49.891921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.892022 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.892043 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:08:49.892091 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 20:08:49.892150 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20251025 20:08:49.892868 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:49.889839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.18.82.124:44491)
1025 20:08:49.890056 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:49.890060 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:49.890069 (+ 9us) server_negotiation.cc:408] Connection header received
1025 20:08:49.890106 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:49.890108 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:49.890151 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:49.890233 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:49.890571 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.891225 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:49.892022 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:49.892496 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:49.892633 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:49.892710 (+ 77us) server_negotiation.cc:300] Negotiation successful
1025 20:08:49.892762 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20251025 20:08:49.912727 25840 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:50.135852 25976 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:50.207319 26324 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251025 20:08:50.207765 26353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cfb2914af2435b90f27010e9d4d8e6. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:50.207906 26353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: LEARNER
I20251025 20:08:50.715117 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: NON_VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: true } } }
I20251025 20:08:50.715281 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:50.715976 26366 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: attempting to promote NON_VOTER 0ec99364029d468c9bc52518dce6b3ad to VOTER
I20251025 20:08:50.716315 26366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:50.717324 26324 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 LEARNER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251025 20:08:50.717468 26139 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Refusing update from remote peer 26cfb2914af2435b90f27010e9d4d8e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251025 20:08:50.717715 26366 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251025 20:08:50.717898 26368 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251025 20:08:50.719393 26368 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.719872 26139 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.721136 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.721223 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:50.721311 26370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.721400 26370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:50.721397 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.721573 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:50.722028 26366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.722126 26366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:50.722404 26324 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.723795 26371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cfb2914af2435b90f27010e9d4d8e6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:50.723955 26371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:50.812553 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.803797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.18.82.125:45489)
1025 20:08:50.804023 (+ 226us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.804027 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.804056 (+ 29us) server_negotiation.cc:408] Connection header received
1025 20:08:50.804156 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.804158 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.804197 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.804264 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:50.804947 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.805449 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.806010 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.806128 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.809872 (+ 3744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:50.809887 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:50.809889 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:50.809912 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:50.811308 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:50.811723 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:50.811725 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:50.811726 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:50.811756 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:50.811927 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:50.811929 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:50.811929 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:50.812064 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:50.812148 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.812288 (+ 140us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.812371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20251025 20:08:50.815394 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.813268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33924)
1025 20:08:50.813487 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.813499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.813605 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.813805 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.813807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.813821 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.813991 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.813995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.814646 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.814648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.815140 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.815146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.815222 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.815235 (+ 13us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.815267 (+ 32us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.815304 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20251025 20:08:50.815747 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.813375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.18.82.126:45153)
1025 20:08:50.813621 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.813624 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.813637 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:50.813676 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.813678 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.813719 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.813786 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.814064 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.814560 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.815216 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.815501 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.815539 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.815599 (+ 60us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.815658 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20251025 20:08:50.818451 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.816656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42756)
1025 20:08:50.816763 (+ 107us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.816773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.816827 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.816998 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.816999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.817024 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.817149 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.817153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.817749 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.817751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.818197 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.818203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.818276 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.818287 (+ 11us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.818344 (+ 57us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.818380 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"spinlock_wait_cycles":13184}
I20251025 20:08:50.818774 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.816727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.18.82.125:45489)
1025 20:08:50.816823 (+ 96us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.816827 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.816837 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:08:50.816884 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.816886 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.816921 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.816996 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.817227 (+ 231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.817681 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.818272 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.818612 (+ 340us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.818637 (+ 25us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.818671 (+ 34us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.818710 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251025 20:08:50.821707 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.819669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59574)
1025 20:08:50.819768 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.819778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.819853 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.820183 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.820185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.820196 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.820338 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.820343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.820957 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.820959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.821430 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.821447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.821553 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.821565 (+ 12us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.821607 (+ 42us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.821643 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251025 20:08:50.822074 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.819777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.18.82.124:44491)
1025 20:08:50.819997 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.820001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.820012 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:50.820051 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.820053 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.820093 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.820160 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.820429 (+ 269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.820878 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.821542 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.821843 (+ 301us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.821882 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.821930 (+ 48us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.821972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20251025 20:08:50.828550 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.823552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.18.82.124:44491)
1025 20:08:50.823657 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.823660 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.823719 (+ 59us) server_negotiation.cc:408] Connection header received
1025 20:08:50.823814 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.823817 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.823850 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.823928 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:50.824469 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.824852 (+ 383us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.825285 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.825385 (+ 100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.826259 (+ 874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:50.826270 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:50.826272 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:50.826297 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:50.827641 (+ 1344us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:50.827918 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:50.827920 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:50.827921 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:50.827949 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:50.828068 (+ 119us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:50.828070 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:50.828070 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:50.828163 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:50.828262 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.828396 (+ 134us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.828469 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251025 20:08:50.831329 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.829169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:33932)
1025 20:08:50.829381 (+ 212us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.829395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.829482 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.829713 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.829715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.829732 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.829889 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.829894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.830504 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.830507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.831006 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.831013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.831112 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.831128 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.831173 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.831222 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20251025 20:08:50.831753 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.829261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.18.82.126:45153)
1025 20:08:50.829379 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.829383 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.829453 (+ 70us) server_negotiation.cc:408] Connection header received
1025 20:08:50.829605 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.829607 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.829642 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.829698 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.829991 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.830405 (+ 414us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.831117 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.831553 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.831580 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.831626 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.831662 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:50.834712 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.832721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.0.0.1:42770)
1025 20:08:50.832825 (+ 104us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.832836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.832894 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.833079 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.833080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.833093 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.833217 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.833221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.833874 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.833877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.834455 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.834463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.834544 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.834556 (+ 12us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.834611 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.834647 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251025 20:08:50.835093 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.832816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.18.82.125:45489)
1025 20:08:50.832928 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.832931 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.832942 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:50.832974 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.832976 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.833009 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.833068 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.833292 (+ 224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.833772 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.834550 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.834900 (+ 350us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.834936 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.834977 (+ 41us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.835015 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 20:08:50.837730 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.835977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.0.0.1:59588)
1025 20:08:50.836087 (+ 110us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.836097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.836149 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.836324 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.836326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.836340 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:50.836455 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.836459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.836984 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.836986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.837447 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.837453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.837567 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.837580 (+ 13us) client_negotiation.cc:769] Sending connection context
1025 20:08:50.837625 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:50.837663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251025 20:08:50.838024 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.836036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.18.82.124:44491)
1025 20:08:50.836122 (+ 86us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.836125 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.836134 (+ 9us) server_negotiation.cc:408] Connection header received
1025 20:08:50.836218 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.836220 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.836251 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.836299 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:50.836522 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.836906 (+ 384us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.837558 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.837828 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:50.837859 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:50.837903 (+ 44us) server_negotiation.cc:300] Negotiation successful
1025 20:08:50.837951 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251025 20:08:50.839123 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25575
I20251025 20:08:50.845916 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:36767
--local_ip_for_outbound_sockets=127.18.82.65
--tserver_master_addrs=127.18.82.126:45153
--webserver_port=35983
--webserver_interface=127.18.82.65
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:50.943181 26378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:50.943372 26378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:50.943437 26378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:50.946305 26378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:50.946383 26378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:50.946470 26378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:50.950083 26378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:36767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=35983
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:50.950944 26378 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:50.951645 26378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:50.957468 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:50.957556 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:50.957610 26378 server_base.cc:1047] running on GCE node
W20251025 20:08:50.957739 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:50.958037 26378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:50.958503 26378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:50.959651 26378 hybrid_clock.cc:648] HybridClock initialized: now 1761422930959637 us; error 34 us; skew 500 ppm
Oct 25 20:08:50 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422930, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:50.962049 26378 init.cc:377] Logged in from keytab as oryx/127.18.82.65@KRBTEST.COM (short username oryx)
I20251025 20:08:50.962958 26378 webserver.cc:492] Webserver started at http://127.18.82.65:35983/ using document root <none> and password file <none>
I20251025 20:08:50.963467 26378 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:50.963534 26378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:50.966670 26378 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:08:50.968243 26393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:50.969061 26378 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:50.969154 26378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "9719aa0b9f60432783f766daf87752fd"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "1bf29b0ce2e90231053cdf43b24e00e5"
server_key_iv: "ecf84b95fa027fdd843db95cd232143d"
server_key_version: "encryptionkey@0"
I20251025 20:08:50.969472 26378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:50.980558 26378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:50.983158 26378 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:50.983316 26378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:50.983777 26378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:50.984560 26378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:50.984608 26378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:50.984680 26378 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:50.984705 26378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:50.992945 26378 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:36767
I20251025 20:08:50.992973 26506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:36767 every 8 connection(s)
I20251025 20:08:50.993871 26378 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:51.001314 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 26378
I20251025 20:08:51.001513 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25711
Oct 25 20:08:50 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422930, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
Oct 25 20:08:50 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422930, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
Oct 25 20:08:50 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422930, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
I20251025 20:08:51.007272 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.996008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:51031 (local address 127.18.82.124:44491)
1025 20:08:50.996115 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.996119 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.996859 (+ 740us) server_negotiation.cc:408] Connection header received
1025 20:08:50.997275 (+ 416us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.997278 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.997329 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.997377 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:50.998587 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999061 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999801 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999970 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.002936 (+ 2966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.002961 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.002968 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.003002 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.004955 (+ 1953us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005543 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.005549 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.005553 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.005618 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005953 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.005959 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.005963 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.006137 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.006216 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.006894 (+ 678us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.007028 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:51.007565 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.995409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:33289 (local address 127.18.82.126:45153)
1025 20:08:50.995521 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.995524 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.996451 (+ 927us) server_negotiation.cc:408] Connection header received
1025 20:08:50.997274 (+ 823us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.997278 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.997329 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.997383 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:50.998679 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999343 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999960 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.000117 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.003268 (+ 3151us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.003287 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.003291 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.003320 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.005664 (+ 2344us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006262 (+ 598us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.006265 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.006267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.006354 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006651 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.006654 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.006656 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.006814 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.006885 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.007263 (+ 378us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.007388 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:51.007565 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.995409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:49697 (local address 127.18.82.125:45489)
1025 20:08:50.995596 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:50.995600 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:50.996628 (+ 1028us) server_negotiation.cc:408] Connection header received
1025 20:08:50.997248 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:50.997251 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:50.997310 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:50.997376 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:50.998558 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999205 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999878 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.000052 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.003653 (+ 3601us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.003675 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.003679 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.003710 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.005664 (+ 1954us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006262 (+ 598us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.006265 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.006267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.006360 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006642 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.006646 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.006647 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.006822 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.006885 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.007263 (+ 378us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.007387 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20251025 20:08:51.008795 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.995903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.18.82.65:51031)
1025 20:08:50.996781 (+ 878us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.996790 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.997098 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.997544 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.997554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.997934 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:50.998443 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.998453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999172 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.999677 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.999686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.000083 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.000650 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.000664 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.002774 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.005082 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.005102 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.005411 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.005745 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.005754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.005859 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.006310 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.006318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.006608 (+ 290us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.007931 (+ 1323us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.008139 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":58,"spinlock_wait_cycles":4224,"thread_start_us":70,"threads_started":1}
I20251025 20:08:51.008785 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.995715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:33289)
1025 20:08:50.996322 (+ 607us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.996366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.997087 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.997544 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.997554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.998328 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:50.998576 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.998582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999473 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.999835 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.999845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.000242 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.000780 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.000791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.003147 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.005798 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.005803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.006153 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.006470 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.006476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.006539 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.006960 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.006963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.007056 (+ 93us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.007732 (+ 676us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.007995 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"spinlock_wait_cycles":26368,"thread_start_us":114,"threads_started":1}
I20251025 20:08:51.008785 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:50.995715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.18.82.65:49697)
1025 20:08:50.996540 (+ 825us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:50.996550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:50.997087 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:50.997545 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:50.997554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:50.997917 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:50.998427 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.998438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999304 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:50.999306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:50.999773 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:50.999783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:50.999947 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.000521 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.000554 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.003504 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.005798 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.005801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.005803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.006153 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.006470 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.006474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.006476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.006539 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.006960 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.006963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.007056 (+ 93us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.007862 (+ 806us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.008060 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":136,"spinlock_wait_cycles":16896,"thread_start_us":94,"threads_started":1}
I20251025 20:08:51.010113 26507 heartbeater.cc:344] Connected to a master server at 127.18.82.124:44491
I20251025 20:08:51.010273 26509 heartbeater.cc:344] Connected to a master server at 127.18.82.125:45489
I20251025 20:08:51.010337 26507 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.010337 26509 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.010901 26507 heartbeater.cc:507] Master 127.18.82.124:44491 requested a full tablet report, sending...
I20251025 20:08:51.010901 26509 heartbeater.cc:507] Master 127.18.82.125:45489 requested a full tablet report, sending...
I20251025 20:08:51.011317 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:42231
--local_ip_for_outbound_sockets=127.18.82.66
--tserver_master_addrs=127.18.82.126:45153
--webserver_port=38477
--webserver_interface=127.18.82.66
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 20:08:51.010919 26508 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:51.011456 26508 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.011644 26508 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:51.012378 26314 ts_manager.cc:194] Registered new tserver with Master: 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
I20251025 20:08:51.012378 26121 ts_manager.cc:194] Registered new tserver with Master: 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
I20251025 20:08:51.012403 25511 ts_manager.cc:194] Re-registered known tserver with Master: 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
I20251025 20:08:51.013077 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:33289
W20251025 20:08:51.110448 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:51.110643 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:51.110724 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:51.113693 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:51.113747 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:51.113866 26514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:51.117722 26514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:42231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=38477
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:51.118738 26514 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:51.119611 26514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:51.125695 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:51.125689 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:51.125689 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:51.125830 26514 server_base.cc:1047] running on GCE node
I20251025 20:08:51.126191 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:51.126725 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:51.127883 26514 hybrid_clock.cc:648] HybridClock initialized: now 1761422931127869 us; error 29 us; skew 500 ppm
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:51.130172 26514 init.cc:377] Logged in from keytab as oryx/127.18.82.66@KRBTEST.COM (short username oryx)
I20251025 20:08:51.130968 26514 webserver.cc:492] Webserver started at http://127.18.82.66:38477/ using document root <none> and password file <none>
I20251025 20:08:51.131423 26514 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:51.131474 26514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:51.134581 26514 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20251025 20:08:51.136191 26529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.137050 26514 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:51.137140 26514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "9b446012218f4316a16864099e5bdbaf"
format_stamp: "Formatted at 2025-10-25 20:08:48 on dist-test-slave-d2pk"
server_key: "2e1cb1ad0610db0a82a9a331690fa8e3"
server_key_iv: "7aae2bf4b1d62571487deae10b491f6c"
server_key_version: "encryptionkey@0"
I20251025 20:08:51.137573 26514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:51.155956 26514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:51.158288 26514 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:51.158408 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:51.158814 26514 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:51.159525 26514 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:51.159569 26514 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.159611 26514 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:51.159631 26514 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.167737 26514 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:42231
I20251025 20:08:51.167784 26642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:42231 every 8 connection(s)
I20251025 20:08:51.168663 26514 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:51.177361 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 26514
I20251025 20:08:51.177538 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25847
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
I20251025 20:08:51.182863 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:55569 (local address 127.18.82.125:45489)
1025 20:08:51.170912 (+ 232us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.170917 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.171540 (+ 623us) server_negotiation.cc:408] Connection header received
1025 20:08:51.172152 (+ 612us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.172156 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.172215 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.172283 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.173758 (+ 1475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.174524 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.175612 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175838 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.178834 (+ 2996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.178853 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.178856 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.178883 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.180767 (+ 1884us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181319 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181322 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181324 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.181376 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181728 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181731 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181733 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.181896 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.181963 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.182570 (+ 607us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.182706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20251025 20:08:51.182863 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:34025 (local address 127.18.82.124:44491)
1025 20:08:51.170997 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.171001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.171698 (+ 697us) server_negotiation.cc:408] Connection header received
1025 20:08:51.172177 (+ 479us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.172181 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.172232 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.172279 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.174684 (+ 2405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175155 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.175904 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.176084 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.178701 (+ 2617us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.178718 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.178721 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.178748 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.180615 (+ 1867us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181306 (+ 691us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181310 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181312 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.181359 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181728 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181731 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181733 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.181896 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.181963 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.182570 (+ 607us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.182706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20251025 20:08:51.182947 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:55799 (local address 127.18.82.126:45153)
1025 20:08:51.170295 (+ 96us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.170299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.171329 (+ 1030us) server_negotiation.cc:408] Connection header received
1025 20:08:51.172178 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.172181 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.172232 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.172279 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.173809 (+ 1530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.174569 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.175323 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175504 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.178049 (+ 2545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.178071 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.178077 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.178107 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.180732 (+ 2625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181434 (+ 702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181441 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181445 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.181500 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181817 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.181823 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.181827 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.182005 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.182084 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.182570 (+ 486us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.182718 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251025 20:08:51.184144 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:55799)
1025 20:08:51.171210 (+ 644us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.171245 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.171982 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.172441 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.172448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.172860 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.173617 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.173634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.174650 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.174653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.175215 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.175223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175404 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.176079 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.176120 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.177893 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.180921 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.180927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.180973 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181308 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.181648 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.181658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181720 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.182146 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.182151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.182307 (+ 156us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.183149 (+ 842us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.183424 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":108,"spinlock_wait_cycles":21888,"thread_start_us":107,"threads_started":1}
I20251025 20:08:51.184130 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.18.82.66:55569)
1025 20:08:51.171454 (+ 889us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.171464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.171982 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.172439 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.172447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.172833 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.173618 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.173633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.174642 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.174645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.175217 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.175223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175404 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.176161 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.176181 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.178717 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.180876 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.180879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.180881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181217 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.181475 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.181479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181617 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.182058 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.182064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.182310 (+ 246us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.183506 (+ 1196us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.183627 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":38,"spinlock_wait_cycles":42752,"thread_start_us":86,"threads_started":1}
I20251025 20:08:51.184130 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.170565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.18.82.66:34025)
1025 20:08:51.171632 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.171641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.171993 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.172441 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.172448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.172860 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.173617 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.173634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.175256 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.175260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.175768 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.175778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.176181 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.176622 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.176634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.178556 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.180753 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.180759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.180768 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181189 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.181473 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.181476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.181478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.181617 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.182058 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.182064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.182310 (+ 246us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.183011 (+ 701us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.183319 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":110,"spinlock_wait_cycles":31744,"thread_start_us":88,"threads_started":1}
I20251025 20:08:51.185653 26645 heartbeater.cc:344] Connected to a master server at 127.18.82.125:45489
I20251025 20:08:51.185870 26645 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.186354 26645 heartbeater.cc:507] Master 127.18.82.125:45489 requested a full tablet report, sending...
I20251025 20:08:51.187000 26644 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:51.187079 26644 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.187284 26644 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:51.187436 26121 ts_manager.cc:194] Registered new tserver with Master: 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:51.187484 26643 heartbeater.cc:344] Connected to a master server at 127.18.82.124:44491
I20251025 20:08:51.187552 26643 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.187743 26643 heartbeater.cc:507] Master 127.18.82.124:44491 requested a full tablet report, sending...
I20251025 20:08:51.188021 25511 ts_manager.cc:194] Re-registered known tserver with Master: 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:51.188400 26314 ts_manager.cc:194] Registered new tserver with Master: 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:51.188390 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:38011
--local_ip_for_outbound_sockets=127.18.82.67
--tserver_master_addrs=127.18.82.126:45153
--webserver_port=43885
--webserver_interface=127.18.82.67
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491 with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 20:08:51.188704 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:55799
W20251025 20:08:51.284428 26650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:51.284616 26650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:51.284691 26650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:51.287520 26650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:51.287590 26650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:51.287681 26650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:51.291574 26650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:38011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=43885
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:45153,127.18.82.125:45489,127.18.82.124:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:51.292439 26650 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:51.293206 26650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:51.299072 26658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:51.299113 26656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:51.299081 26655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:51.299211 26650 server_base.cc:1047] running on GCE node
I20251025 20:08:51.299647 26650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:51.300107 26650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:51.301244 26650 hybrid_clock.cc:648] HybridClock initialized: now 1761422931301229 us; error 26 us; skew 500 ppm
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:51.303992 26650 init.cc:377] Logged in from keytab as oryx/127.18.82.67@KRBTEST.COM (short username oryx)
I20251025 20:08:51.304874 26650 webserver.cc:492] Webserver started at http://127.18.82.67:43885/ using document root <none> and password file <none>
I20251025 20:08:51.305357 26650 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:51.305423 26650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:51.308614 26650 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:51.310252 26665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.311134 26650 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:51.311235 26650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "14185135b8d443c2aeecd6d5717b930f"
format_stamp: "Formatted at 2025-10-25 20:08:49 on dist-test-slave-d2pk"
server_key: "9ae9e322861348ddc4efbc74bdeade8c"
server_key_iv: "9986af2d5d3c85223c17540b76921431"
server_key_version: "encryptionkey@0"
I20251025 20:08:51.311602 26650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:51.336726 26650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:51.339404 26650 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:51.339567 26650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:51.340106 26650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:08:51.340989 26650 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:51.341034 26650 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.341091 26650 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:51.341116 26650 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:51.349177 26650 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:38011
I20251025 20:08:51.349212 26778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:38011 every 8 connection(s)
I20251025 20:08:51.350099 26650 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:51.353596 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 26650
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.125@KRBTEST.COM
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.124@KRBTEST.COM
Oct 25 20:08:51 dist-test-slave-d2pk krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422931, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:51.365211 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.351581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:35421 (local address 127.18.82.125:45489)
1025 20:08:51.351714 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.351718 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.352431 (+ 713us) server_negotiation.cc:408] Connection header received
1025 20:08:51.353540 (+ 1109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.353544 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.353601 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.353676 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.355294 (+ 1618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.355915 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.356723 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.356924 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.360080 (+ 3156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.360097 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.360100 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.360124 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.363087 (+ 2963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.363682 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.363686 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.363688 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.363736 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.364077 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.364080 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.364083 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.364229 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.364330 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.364943 (+ 613us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.365050 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 20:08:51.366436 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.351849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.125:45489 (local address 127.18.82.67:35421)
1025 20:08:51.352296 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.352345 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.353204 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.353859 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.353870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.354430 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.355132 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.355150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.356051 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.356054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.356601 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.356611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.357121 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.357729 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.357777 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.359734 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.363222 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.363229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.363243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.363556 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.363838 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.363842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.363844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.363972 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.364390 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.364396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.364652 (+ 256us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.365362 (+ 710us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.365656 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":4352,"thread_start_us":116,"threads_started":1}
I20251025 20:08:51.366722 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.351850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.124:44491 (local address 127.18.82.67:49331)
1025 20:08:51.352961 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.352975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.353227 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.354337 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.354340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.354518 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.355233 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.355242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.356136 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.356139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.356724 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.356732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.357350 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.358309 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.358340 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.360669 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.364952 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.364955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.364956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.365135 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.365698 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.365703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.365710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.365770 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.366206 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.366210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.366319 (+ 109us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.366427 (+ 108us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.366542 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":749,"spinlock_wait_cycles":2432,"thread_start_us":97,"threads_started":1}
I20251025 20:08:51.367197 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.357869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.18.82.124:44491)
1025 20:08:51.359947 (+ 2078us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.359950 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.359961 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:08:51.359991 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.359994 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.360034 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.360105 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.361012 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.361663 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.362630 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.362826 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.363688 (+ 862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.363706 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.363709 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.363732 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.365516 (+ 1784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.365934 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.365938 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.365940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.365984 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.366196 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.366199 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.366201 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.366366 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.366481 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.366971 (+ 490us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.367079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2003,"thread_start_us":78,"threads_started":1}
I20251025 20:08:51.367436 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.352049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:49331 (local address 127.18.82.124:44491)
1025 20:08:51.352356 (+ 307us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.352360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.353038 (+ 678us) server_negotiation.cc:408] Connection header received
1025 20:08:51.354097 (+ 1059us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.354101 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.354157 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.354228 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.355355 (+ 1127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.355989 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.357062 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.357240 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.360796 (+ 3556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.360814 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.360819 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.360865 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.364256 (+ 3391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.365225 (+ 969us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.365228 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.365230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.365270 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.365882 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.365886 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.365887 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.366063 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.366159 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.367175 (+ 1016us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.367281 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20251025 20:08:51.367725 26781 heartbeater.cc:344] Connected to a master server at 127.18.82.125:45489
I20251025 20:08:51.367949 26781 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.367988 26779 heartbeater.cc:344] Connected to a master server at 127.18.82.124:44491
I20251025 20:08:51.368046 26779 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.368498 26779 heartbeater.cc:507] Master 127.18.82.124:44491 requested a full tablet report, sending...
I20251025 20:08:51.368498 26781 heartbeater.cc:507] Master 127.18.82.125:45489 requested a full tablet report, sending...
I20251025 20:08:51.369478 26314 ts_manager.cc:194] Registered new tserver with Master: 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011)
I20251025 20:08:51.369500 26121 ts_manager.cc:194] Registered new tserver with Master: 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011)
I20251025 20:08:51.370069 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.355262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.18.82.126:45153)
1025 20:08:51.356409 (+ 1147us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.356414 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.356715 (+ 301us) server_negotiation.cc:408] Connection header received
1025 20:08:51.356752 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.356755 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.356799 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.356885 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.358717 (+ 1832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.359491 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.363437 (+ 3946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.363657 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.365204 (+ 1547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.365220 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.365223 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.365245 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.367596 (+ 2351us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.368811 (+ 1215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.368814 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.368816 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.368859 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.369098 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.369104 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.369108 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.369262 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.369380 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.369830 (+ 450us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.369936 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":85,"threads_started":1}
I20251025 20:08:51.370473 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.352470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:41503 (local address 127.18.82.126:45153)
1025 20:08:51.352598 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.352602 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.352986 (+ 384us) server_negotiation.cc:408] Connection header received
1025 20:08:51.353626 (+ 640us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.353630 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.353695 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.353775 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.355319 (+ 1544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.355997 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.356869 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.357061 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.361434 (+ 4373us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.361453 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.361458 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.361491 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.364831 (+ 3340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.369174 (+ 4343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.369177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.369179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.369228 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.369468 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.369471 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.369472 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.369644 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.369723 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.370296 (+ 573us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.370388 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:08:51.370721 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.351850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:41503)
1025 20:08:51.352886 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.352901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.353227 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.353859 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.353870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.354464 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:51.355170 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.355178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.356136 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.356139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.356724 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.356732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.357172 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.358077 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:51.358093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:51.361306 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:51.368895 (+ 7589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.368898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.368899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.369071 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.369318 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:51.369321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:51.369322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:51.369379 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:51.369848 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:51.369853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:51.369953 (+ 100us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.370038 (+ 85us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.370610 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":43,"spinlock_wait_cycles":12416,"thread_start_us":91,"threads_started":1}
I20251025 20:08:51.371344 26780 heartbeater.cc:344] Connected to a master server at 127.18.82.126:45153
I20251025 20:08:51.371426 26780 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:51.371588 26780 heartbeater.cc:507] Master 127.18.82.126:45153 requested a full tablet report, sending...
I20251025 20:08:51.372169 25511 ts_manager.cc:194] Re-registered known tserver with Master: 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011)
I20251025 20:08:51.373589 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.355897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.18.82.125:45489)
1025 20:08:51.357700 (+ 1803us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.357707 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.358452 (+ 745us) server_negotiation.cc:408] Connection header received
1025 20:08:51.358495 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.358498 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.358542 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.358631 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.359635 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.360369 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.361271 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.361433 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.362422 (+ 989us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.362438 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.362441 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.362468 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.372064 (+ 9596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.372481 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.372484 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.372486 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.372532 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.372831 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.372834 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.372835 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.372986 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.373086 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.373287 (+ 201us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.373388 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1720,"thread_start_us":68,"threads_started":1}
I20251025 20:08:51.373865 25511 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:41503
I20251025 20:08:51.372546 25510 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:51.375392 25510 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:51.389259 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.385459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:38011 (local address 127.0.0.1:60068)
1025 20:08:51.385814 (+ 355us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.385829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.385909 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.386654 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.386660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.386687 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:51.386897 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.386904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.388020 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.388024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.388859 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.388869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.388965 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.388982 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.389022 (+ 40us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.389078 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
I20251025 20:08:51.389863 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.385886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36767 (local address 127.0.0.1:37304)
1025 20:08:51.387062 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.387076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.387161 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.387611 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.387614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.387629 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:51.387835 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.387842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.388758 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.388761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.389467 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.389475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.389609 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.389626 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.389674 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.389723 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":87,"threads_started":1}
I20251025 20:08:51.390491 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.385848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.18.82.67:38011)
1025 20:08:51.386143 (+ 295us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.386152 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.386172 (+ 20us) server_negotiation.cc:408] Connection header received
1025 20:08:51.386264 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.386272 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.386503 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.386671 (+ 168us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:51.387038 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.387920 (+ 882us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.389720 (+ 1800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.390138 (+ 418us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.390243 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.390299 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.390370 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20251025 20:08:51.390756 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.386254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.18.82.65:36767)
1025 20:08:51.386955 (+ 701us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.386963 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.387205 (+ 242us) server_negotiation.cc:408] Connection header received
1025 20:08:51.387302 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.387307 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.387497 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.387634 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:51.387957 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.388658 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.390009 (+ 1351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.390487 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.390553 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.390609 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.390661 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":137,"threads_started":1}
I20251025 20:08:51.390393 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.386382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:42231 (local address 127.0.0.1:43976)
1025 20:08:51.387332 (+ 950us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.387354 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.387433 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.387805 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.387808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.387823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:51.388019 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.388026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.389210 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.389213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.390064 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.390073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.390172 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.390187 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.390229 (+ 42us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.390277 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":82,"threads_started":1}
I20251025 20:08:51.391013 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.386546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.18.82.66:42231)
1025 20:08:51.387159 (+ 613us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.387165 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.387440 (+ 275us) server_negotiation.cc:408] Connection header received
1025 20:08:51.387553 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.387557 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.387699 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.387815 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:51.388126 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.389089 (+ 963us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.390185 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.390659 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.390756 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.390818 (+ 62us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.390887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":97,"threads_started":1}
I20251025 20:08:51.393357 26441 tablet_service.cc:1505] Processing CreateTablet for tablet c341a19e45af4392a17c566d97d9c22d (DEFAULT_TABLE table=test-table [id=9ac86d2fa7744d21bc4bc118bdc53c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:51.393359 26577 tablet_service.cc:1505] Processing CreateTablet for tablet c341a19e45af4392a17c566d97d9c22d (DEFAULT_TABLE table=test-table [id=9ac86d2fa7744d21bc4bc118bdc53c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:51.393365 26713 tablet_service.cc:1505] Processing CreateTablet for tablet c341a19e45af4392a17c566d97d9c22d (DEFAULT_TABLE table=test-table [id=9ac86d2fa7744d21bc4bc118bdc53c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:51.394220 26441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341a19e45af4392a17c566d97d9c22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:51.394232 26577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341a19e45af4392a17c566d97d9c22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:51.394232 26713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341a19e45af4392a17c566d97d9c22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:51.399863 26809 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Bootstrap starting.
I20251025 20:08:51.401257 26810 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Bootstrap starting.
I20251025 20:08:51.402262 26809 tablet_bootstrap.cc:654] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:51.402951 26809 log.cc:826] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:51.403224 26810 tablet_bootstrap.cc:654] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:51.404009 26810 log.cc:826] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:51.404479 26809 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: No bootstrap required, opened a new log
I20251025 20:08:51.404680 26809 ts_tablet_manager.cc:1403] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:08:51.405740 26810 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: No bootstrap required, opened a new log
I20251025 20:08:51.405943 26810 ts_tablet_manager.cc:1403] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:08:51.407215 26809 raft_consensus.cc:359] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.407387 26809 raft_consensus.cc:385] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:51.407477 26809 raft_consensus.cc:740] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9719aa0b9f60432783f766daf87752fd, State: Initialized, Role: FOLLOWER
I20251025 20:08:51.407904 26809 consensus_queue.cc:260] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.408672 26809 ts_tablet_manager.cc:1434] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20251025 20:08:51.409489 26508 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:51.409840 26810 raft_consensus.cc:359] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.410015 26810 raft_consensus.cc:385] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:51.410058 26810 raft_consensus.cc:740] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14185135b8d443c2aeecd6d5717b930f, State: Initialized, Role: FOLLOWER
I20251025 20:08:51.410492 26810 consensus_queue.cc:260] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.411891 26810 ts_tablet_manager.cc:1434] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251025 20:08:51.412272 26780 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:51.415755 26815 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Bootstrap starting.
I20251025 20:08:51.417852 26815 tablet_bootstrap.cc:654] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:51.418486 26815 log.cc:826] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:51.419859 26815 tablet_bootstrap.cc:492] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: No bootstrap required, opened a new log
I20251025 20:08:51.419986 26815 ts_tablet_manager.cc:1403] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 20:08:51.422106 26815 raft_consensus.cc:359] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.422331 26815 raft_consensus.cc:385] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:51.422399 26815 raft_consensus.cc:740] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b446012218f4316a16864099e5bdbaf, State: Initialized, Role: FOLLOWER
I20251025 20:08:51.422812 26815 consensus_queue.cc:260] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.423442 26644 heartbeater.cc:499] Master 127.18.82.126:45153 was elected leader, sending a full tablet report...
I20251025 20:08:51.423552 26815 ts_tablet_manager.cc:1434] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20251025 20:08:51.495632 26510 tablet.cc:2378] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 20:08:51.601634 26782 tablet.cc:2378] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 20:08:51.670877 26646 tablet.cc:2378] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:51.760825 26813 raft_consensus.cc:493] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:51.760998 26813 raft_consensus.cc:515] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.762097 26813 leader_election.cc:290] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:51.764909 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.762317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:42231 (local address 127.18.82.65:52825)
1025 20:08:51.762653 (+ 336us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.762667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.762787 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.763046 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.763049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.763069 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:51.763264 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.763270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.763990 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.763993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.764496 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.764502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.764608 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.764630 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.764675 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.764730 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":61,"threads_started":1}
I20251025 20:08:51.764915 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.762269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:38011 (local address 127.18.82.65:46249)
1025 20:08:51.762565 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:51.762579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:51.762700 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:51.762998 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:51.763003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:51.763021 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:51.763250 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.763257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.763993 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.763995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:51.764506 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:51.764513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.764609 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.764630 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:08:51.764674 (+ 44us) client_negotiation.cc:240] Negotiation successful
1025 20:08:51.764728 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":68,"threads_started":1}
I20251025 20:08:51.765296 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.762434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:52825 (local address 127.18.82.66:42231)
1025 20:08:51.762601 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.762609 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.762745 (+ 136us) server_negotiation.cc:408] Connection header received
1025 20:08:51.762886 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.762888 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.762930 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.763009 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:51.763387 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.763862 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.764605 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.765036 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.765077 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.765141 (+ 64us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.765196 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251025 20:08:51.765327 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.762398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:46249 (local address 127.18.82.67:38011)
1025 20:08:51.762602 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.762607 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.762661 (+ 54us) server_negotiation.cc:408] Connection header received
1025 20:08:51.762827 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.762833 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.762897 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.762958 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:51.763371 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.763853 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.764613 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.765033 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.765077 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.765151 (+ 74us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.765206 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20251025 20:08:51.765843 26597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341a19e45af4392a17c566d97d9c22d" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b446012218f4316a16864099e5bdbaf" is_pre_election: true
I20251025 20:08:51.765852 26733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341a19e45af4392a17c566d97d9c22d" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14185135b8d443c2aeecd6d5717b930f" is_pre_election: true
I20251025 20:08:51.766083 26733 raft_consensus.cc:2468] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 0.
I20251025 20:08:51.766094 26597 raft_consensus.cc:2468] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 0.
I20251025 20:08:51.766506 26396 leader_election.cc:304] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14185135b8d443c2aeecd6d5717b930f, 9719aa0b9f60432783f766daf87752fd; no voters:
I20251025 20:08:51.766741 26813 raft_consensus.cc:2804] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:51.766813 26813 raft_consensus.cc:493] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:51.766844 26813 raft_consensus.cc:3060] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:51.767732 26813 raft_consensus.cc:515] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.768018 26813 leader_election.cc:290] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 election: Requested vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:51.768323 26733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341a19e45af4392a17c566d97d9c22d" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14185135b8d443c2aeecd6d5717b930f"
I20251025 20:08:51.768376 26597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341a19e45af4392a17c566d97d9c22d" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b446012218f4316a16864099e5bdbaf"
I20251025 20:08:51.768440 26733 raft_consensus.cc:3060] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:51.768463 26597 raft_consensus.cc:3060] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:51.769480 26597 raft_consensus.cc:2468] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 1.
I20251025 20:08:51.769480 26733 raft_consensus.cc:2468] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 1.
I20251025 20:08:51.769780 26396 leader_election.cc:304] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14185135b8d443c2aeecd6d5717b930f, 9719aa0b9f60432783f766daf87752fd; no voters:
I20251025 20:08:51.769963 26813 raft_consensus.cc:2804] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:51.770087 26813 raft_consensus.cc:697] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 1 LEADER]: Becoming Leader. State: Replica: 9719aa0b9f60432783f766daf87752fd, State: Running, Role: LEADER
I20251025 20:08:51.770370 26813 consensus_queue.cc:237] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:51.773288 25511 catalog_manager.cc:5649] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9719aa0b9f60432783f766daf87752fd (127.18.82.65). New cstate: current_term: 1 leader_uuid: "9719aa0b9f60432783f766daf87752fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:51.837894 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.835225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.18.82.65:36767)
1025 20:08:51.835373 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.835377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.835404 (+ 27us) server_negotiation.cc:408] Connection header received
1025 20:08:51.835495 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.835497 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.835548 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.835619 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:51.835911 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.836398 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.836905 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.837020 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.837102 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:51.837579 (+ 477us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:51.837669 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.837750 (+ 81us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.837794 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251025 20:08:51.843152 26597 raft_consensus.cc:1275] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Refusing update from remote peer 9719aa0b9f60432783f766daf87752fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:51.843190 26733 raft_consensus.cc:1275] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Refusing update from remote peer 9719aa0b9f60432783f766daf87752fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:51.843816 26813 consensus_queue.cc:1048] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:51.844012 26820 consensus_queue.cc:1048] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:51.852862 26824 mvcc.cc:204] Tried to move back new op lower bound from 7214788328822927360 to 7214788328533524480. Current Snapshot: MvccSnapshot[applied={T|T < 7214788328822927360}]
I20251025 20:08:51.860025 25511 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:51.860168 25511 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33948:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:51.865582 25511 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 26cfb2914af2435b90f27010e9d4d8e6: Sending DeleteTablet for 3 replicas of tablet c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:51.866425 26441 tablet_service.cc:1552] Processing DeleteTablet for tablet c341a19e45af4392a17c566d97d9c22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:51 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37304
I20251025 20:08:51.867203 26713 tablet_service.cc:1552] Processing DeleteTablet for tablet c341a19e45af4392a17c566d97d9c22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:51 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:60068
I20251025 20:08:51.867583 26712 tablet_service.cc:1552] Processing DeleteTablet for tablet c341a19e45af4392a17c566d97d9c22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:51 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:60068
I20251025 20:08:51.867710 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25479
I20251025 20:08:51.868856 26834 tablet_replica.cc:333] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: stopping tablet replica
I20251025 20:08:51.869124 26834 raft_consensus.cc:2243] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:51.869477 26834 raft_consensus.cc:2272] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:51.870954 26834 ts_tablet_manager.cc:1916] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:51.873492 26834 ts_tablet_manager.cc:1929] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:51.873672 26834 log.cc:1199] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:51.874074 26834 ts_tablet_manager.cc:1950] T c341a19e45af4392a17c566d97d9c22d P 14185135b8d443c2aeecd6d5717b930f: Deleting consensus metadata
W20251025 20:08:51.875084 26667 connection.cc:441] server connection from 127.0.0.1:60068 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60068 (request call id 1) could send its response
W20251025 20:08:51.875756 26667 connection.cc:441] server connection from 127.0.0.1:60068 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60068 (request call id 2) could send its response
I20251025 20:08:51.877091 26833 tablet_replica.cc:333] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: stopping tablet replica
I20251025 20:08:51.877406 26833 raft_consensus.cc:2243] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:51.877868 26833 raft_consensus.cc:2272] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:51.879305 26833 ts_tablet_manager.cc:1916] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:51.881551 26833 ts_tablet_manager.cc:1929] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:51.881692 26833 log.cc:1199] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:51.882100 26833 ts_tablet_manager.cc:1950] T c341a19e45af4392a17c566d97d9c22d P 9719aa0b9f60432783f766daf87752fd: Deleting consensus metadata
W20251025 20:08:51.883118 26396 connection.cc:441] server connection from 127.0.0.1:37304 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37304 (request call id 1) could send its response
I20251025 20:08:51.886113 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.877371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.18.82.124:44491)
1025 20:08:51.878185 (+ 814us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.878189 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.878215 (+ 26us) server_negotiation.cc:408] Connection header received
1025 20:08:51.878255 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.878258 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.878310 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.878399 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.879480 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.880174 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.880966 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.881144 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.882439 (+ 1295us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.882457 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.882460 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.882485 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.884432 (+ 1947us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.884815 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.884818 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.884820 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.884860 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.885055 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.885058 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.885060 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.885226 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.885362 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.885855 (+ 493us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.885960 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":55,"threads_started":1}
I20251025 20:08:51.888468 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:51.877097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.18.82.125:45489)
1025 20:08:51.881689 (+ 4592us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:51.881693 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:51.881706 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:51.881745 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:51.881748 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:51.881794 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:51.882698 (+ 904us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:51.882839 (+ 141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.883738 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:51.884296 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:51.884491 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:51.885351 (+ 860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:51.885365 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:51.885368 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:51.885391 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:51.887186 (+ 1795us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.887565 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.887568 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.887569 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.887613 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:51.887796 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:51.887799 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:51.887801 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:51.887923 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:51.888030 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:51.888210 (+ 180us) server_negotiation.cc:300] Negotiation successful
1025 20:08:51.888313 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4510,"thread_start_us":56,"threads_started":1}
I20251025 20:08:52.855186 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:52.854743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:41215)
1025 20:08:52.854998 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:52.855078 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
W20251025 20:08:52.855620 26644 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:45153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:52.859067 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:52.858698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:38175)
1025 20:08:52.858917 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:52.858989 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
W20251025 20:08:52.859311 26508 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:45153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:52.869048 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:52.868752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:33805)
1025 20:08:52.868929 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:52.868981 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
W20251025 20:08:52.869259 26780 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:45153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.352192 26847 raft_consensus.cc:493] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9719aa0b9f60432783f766daf87752fd)
I20251025 20:08:53.352319 26847 raft_consensus.cc:515] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:53.353349 26847 leader_election.cc:290] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
I20251025 20:08:53.356060 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.353594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:38011 (local address 127.18.82.66:44105)
1025 20:08:53.353731 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.353747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:53.353865 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:53.354223 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:53.354227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:53.354243 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:53.354443 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:53.354448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.355142 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:53.355146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:53.355665 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:53.355673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.355778 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:53.355801 (+ 23us) client_negotiation.cc:769] Sending connection context
1025 20:08:53.355849 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 20:08:53.355929 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251025 20:08:53.356549 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.353600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36767 (local address 127.18.82.66:38143)
1025 20:08:53.354107 (+ 507us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.354117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:53.354222 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:53.354726 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:53.354728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:53.354747 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:53.354968 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:53.354974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.355627 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:53.355630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:53.356221 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:53.356230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.356346 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:53.356364 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:53.356404 (+ 40us) client_negotiation.cc:240] Negotiation successful
1025 20:08:53.356448 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":250,"thread_start_us":58,"threads_started":1}
I20251025 20:08:53.356559 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.353699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:44105 (local address 127.18.82.67:38011)
1025 20:08:53.353966 (+ 267us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:53.353971 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:53.353987 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:08:53.354042 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:53.354046 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:53.354100 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:53.354186 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:53.354529 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.355036 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:53.355789 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.356271 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:53.356316 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:53.356377 (+ 61us) server_negotiation.cc:300] Negotiation successful
1025 20:08:53.356436 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20251025 20:08:53.357220 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.353745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:38143 (local address 127.18.82.65:36767)
1025 20:08:53.354025 (+ 280us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:53.354030 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:53.354533 (+ 503us) server_negotiation.cc:408] Connection header received
1025 20:08:53.354587 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:53.354590 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:53.354634 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:53.354699 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:53.355085 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.355521 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:53.356334 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:53.356944 (+ 610us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:53.356976 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:53.357018 (+ 42us) server_negotiation.cc:300] Negotiation successful
1025 20:08:53.357069 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
W20251025 20:08:53.357321 26532 leader_election.cc:343] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011): Not found: Tablet not found: c341a19e45af4392a17c566d97d9c22d
W20251025 20:08:53.358027 26530 leader_election.cc:343] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767): Not found: Tablet not found: c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:53.358108 26530 leader_election.cc:304] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b446012218f4316a16864099e5bdbaf; no voters: 14185135b8d443c2aeecd6d5717b930f, 9719aa0b9f60432783f766daf87752fd
I20251025 20:08:53.358335 26847 raft_consensus.cc:2749] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:08:53.365661 26851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26cfb2914af2435b90f27010e9d4d8e6)
I20251025 20:08:53.365774 26852 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26cfb2914af2435b90f27010e9d4d8e6)
I20251025 20:08:53.365839 26851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:53.365938 26852 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:53.366787 26852 leader_election.cc:290] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125:45489)
I20251025 20:08:53.366806 26851 leader_election.cc:290] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124:44491)
I20251025 20:08:53.367297 26139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec99364029d468c9bc52518dce6b3ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" is_pre_election: true
I20251025 20:08:53.367296 26324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ec99364029d468c9bc52518dce6b3ad" is_pre_election: true
I20251025 20:08:53.367429 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.366934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37694)
1025 20:08:53.367208 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.367280 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":56,"threads_started":1}
I20251025 20:08:53.367483 26324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e87b1cb0ca4b91aec6bc3ff3426fe4 in term 1.
I20251025 20:08:53.367483 26139 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec99364029d468c9bc52518dce6b3ad in term 1.
I20251025 20:08:53.367530 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.367104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37704)
1025 20:08:53.367371 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.367432 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":56,"threads_started":1}
W20251025 20:08:53.367568 26116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
W20251025 20:08:53.367635 26301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.367810 26113 leader_election.cc:304] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec99364029d468c9bc52518dce6b3ad, 83e87b1cb0ca4b91aec6bc3ff3426fe4; no voters:
I20251025 20:08:53.367810 26299 leader_election.cc:304] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec99364029d468c9bc52518dce6b3ad, 83e87b1cb0ca4b91aec6bc3ff3426fe4; no voters:
I20251025 20:08:53.368005 26852 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:08:53.368081 26852 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26cfb2914af2435b90f27010e9d4d8e6)
I20251025 20:08:53.368139 26852 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:08:53.368117 26851 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 20:08:53.368185 26851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26cfb2914af2435b90f27010e9d4d8e6)
I20251025 20:08:53.368243 26851 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:08:53.368541 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.368300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37712)
1025 20:08:53.368429 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.368477 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20251025 20:08:53.368659 26116 leader_election.cc:336] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.368804 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.368571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37718)
1025 20:08:53.368709 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.368750 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20251025 20:08:53.368886 26301 leader_election.cc:336] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.369359 26852 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:53.369386 26851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:53.369731 26851 leader_election.cc:290] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 election: Requested vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124:44491)
I20251025 20:08:53.369782 26852 leader_election.cc:290] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 election: Requested vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 83e87b1cb0ca4b91aec6bc3ff3426fe4 (127.18.82.125:45489)
I20251025 20:08:53.370025 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.369781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37722)
1025 20:08:53.369910 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.369971 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20251025 20:08:53.370061 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.369815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37730)
1025 20:08:53.369955 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.369999 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20251025 20:08:53.370090 26139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec99364029d468c9bc52518dce6b3ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4"
I20251025 20:08:53.370201 26324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ec99364029d468c9bc52518dce6b3ad"
I20251025 20:08:53.370240 26139 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ec99364029d468c9bc52518dce6b3ad in current term 2: Already voted for candidate 83e87b1cb0ca4b91aec6bc3ff3426fe4 in this term.
I20251025 20:08:53.370324 26324 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83e87b1cb0ca4b91aec6bc3ff3426fe4 in current term 2: Already voted for candidate 0ec99364029d468c9bc52518dce6b3ad in this term.
I20251025 20:08:53.370776 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.370559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37732)
1025 20:08:53.370688 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.370731 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20251025 20:08:53.370885 26116 leader_election.cc:336] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.370940 26116 leader_election.cc:304] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e87b1cb0ca4b91aec6bc3ff3426fe4; no voters: 0ec99364029d468c9bc52518dce6b3ad, 26cfb2914af2435b90f27010e9d4d8e6
I20251025 20:08:53.371049 26851 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:08:53.371063 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.370823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:37734)
1025 20:08:53.370975 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.371016 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20251025 20:08:53.371138 26301 leader_election.cc:336] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:53.371187 26301 leader_election.cc:304] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec99364029d468c9bc52518dce6b3ad; no voters: 26cfb2914af2435b90f27010e9d4d8e6, 83e87b1cb0ca4b91aec6bc3ff3426fe4
I20251025 20:08:53.371295 26852 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 20:08:53.856521 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.856218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:41511)
1025 20:08:53.856353 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.856425 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251025 20:08:53.860090 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.859724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:45577)
1025 20:08:53.859961 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.860011 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20251025 20:08:53.870048 26858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:53.869714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:55445)
1025 20:08:53.869922 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:53.869969 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20251025 20:08:54.857764 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:54.857276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:41907)
1025 20:08:54.857622 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:54.857680 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":81,"threads_started":1}
I20251025 20:08:54.861028 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:54.860631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:59963)
1025 20:08:54.860890 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:54.860945 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20251025 20:08:54.870898 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:54.870580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:58419)
1025 20:08:54.870780 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:54.870824 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20251025 20:08:55.068599 26862 raft_consensus.cc:493] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9719aa0b9f60432783f766daf87752fd)
I20251025 20:08:55.068729 26862 raft_consensus.cc:515] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.069151 26862 leader_election.cc:290] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767)
W20251025 20:08:55.070019 26530 leader_election.cc:343] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767): Not found: Tablet not found: c341a19e45af4392a17c566d97d9c22d
W20251025 20:08:55.070119 26532 leader_election.cc:343] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011): Not found: Tablet not found: c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:55.070226 26532 leader_election.cc:304] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b446012218f4316a16864099e5bdbaf; no voters: 14185135b8d443c2aeecd6d5717b930f, 9719aa0b9f60432783f766daf87752fd
I20251025 20:08:55.070394 26862 raft_consensus.cc:2749] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 20:08:55.258611 26863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:55.258740 26863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:55.259136 26863 leader_election.cc:290] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124:44491)
I20251025 20:08:55.259589 26324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ec99364029d468c9bc52518dce6b3ad" is_pre_election: true
I20251025 20:08:55.259704 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.259327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38006)
1025 20:08:55.259555 (+ 228us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.259616 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":55,"threads_started":1}
I20251025 20:08:55.259733 26324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e87b1cb0ca4b91aec6bc3ff3426fe4 in term 2.
I20251025 20:08:55.260076 26113 leader_election.cc:304] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec99364029d468c9bc52518dce6b3ad, 83e87b1cb0ca4b91aec6bc3ff3426fe4; no voters:
I20251025 20:08:55.260260 26863 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 20:08:55.260320 26863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:55.260421 26863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:08:55.260584 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.260323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38020)
1025 20:08:55.260484 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.260526 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20251025 20:08:55.260710 26116 leader_election.cc:336] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:55.261639 26863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:55.261968 26863 leader_election.cc:290] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 election: Requested vote from peers 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153), 0ec99364029d468c9bc52518dce6b3ad (127.18.82.124:44491)
I20251025 20:08:55.262279 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.262081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38036)
1025 20:08:55.262173 (+ 92us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.262216 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20251025 20:08:55.262362 26324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ec99364029d468c9bc52518dce6b3ad"
I20251025 20:08:55.262491 26324 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 2 FOLLOWER]: Advancing to term 3
I20251025 20:08:55.262956 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.262749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38052)
1025 20:08:55.262857 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.262902 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251025 20:08:55.263067 26116 leader_election.cc:336] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
I20251025 20:08:55.263569 26324 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e87b1cb0ca4b91aec6bc3ff3426fe4 in term 3.
I20251025 20:08:55.263847 26113 leader_election.cc:304] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec99364029d468c9bc52518dce6b3ad, 83e87b1cb0ca4b91aec6bc3ff3426fe4; no voters: 26cfb2914af2435b90f27010e9d4d8e6
I20251025 20:08:55.264001 26863 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 20:08:55.264158 26863 raft_consensus.cc:697] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [term 3 LEADER]: Becoming Leader. State: Replica: 83e87b1cb0ca4b91aec6bc3ff3426fe4, State: Running, Role: LEADER
I20251025 20:08:55.264430 26863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } }
I20251025 20:08:55.265424 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e87b1cb0ca4b91aec6bc3ff3426fe4. Latest consensus state: current_term: 3 leader_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:55.265551 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:55.265859 26868 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:55.267966 26868 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:55.268350 26868 catalog_manager.cc:1269] Loaded cluster ID: 8d84a661b37e46ec9999f3adb8a9e7cb
I20251025 20:08:55.268394 26868 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:55.268913 26868 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:55.269284 26868 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Loaded TSK: 0
I20251025 20:08:55.269985 26868 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:08:55.323978 26121 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:55.326473 26121 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:55.329308 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.329040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38066)
1025 20:08:55.329191 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.329241 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20251025 20:08:55.329362 26324 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [term 3 FOLLOWER]: Refusing update from remote peer 83e87b1cb0ca4b91aec6bc3ff3426fe4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251025 20:08:55.329881 26866 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251025 20:08:55.329983 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.329774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.0.0.1:38078)
1025 20:08:55.329888 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.329929 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251025 20:08:55.330090 26116 consensus_peers.cc:597] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 -> Peer 26cfb2914af2435b90f27010e9d4d8e6 (127.18.82.126:45153): Couldn't send request to peer 26cfb2914af2435b90f27010e9d4d8e6. Status: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 20:08:55.331283 26870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e87b1cb0ca4b91aec6bc3ff3426fe4. Latest consensus state: current_term: 3 leader_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:55.331394 26870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:55.332361 26863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:55.332485 26863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:55.332617 26871 mvcc.cc:204] Tried to move back new op lower bound from 7214788343104798720 to 7214788342845411328. Current Snapshot: MvccSnapshot[applied={T|T < 7214788343104798720}]
I20251025 20:08:55.334728 26870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "26cfb2914af2435b90f27010e9d4d8e6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 45153 } } peers { permanent_uuid: "83e87b1cb0ca4b91aec6bc3ff3426fe4" member_type: VOTER last_known_addr { host: "127.18.82.125" port: 45489 } attrs { promote: false } } peers { permanent_uuid: "0ec99364029d468c9bc52518dce6b3ad" member_type: VOTER last_known_addr { host: "127.18.82.124" port: 44491 } attrs { promote: false } } }
I20251025 20:08:55.334816 26870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec99364029d468c9bc52518dce6b3ad [sys.catalog]: This master's current role is: FOLLOWER
I20251025 20:08:55.341089 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.337999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36767 (local address 127.0.0.1:43634)
1025 20:08:55.338124 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.338139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:55.338239 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:55.338707 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:55.338710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:55.338744 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:55.338936 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.338941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.339860 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.339864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:55.340690 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.340699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.340826 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.340852 (+ 26us) client_negotiation.cc:769] Sending connection context
1025 20:08:55.340902 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 20:08:55.340969 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":5376}
I20251025 20:08:55.342003 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.338351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:38011 (local address 127.0.0.1:40996)
1025 20:08:55.338635 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.338648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:55.338746 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:55.339732 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:55.339735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:55.339750 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:55.339937 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.339945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.340655 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.340658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:55.341572 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.341583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.341704 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.341723 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 20:08:55.341775 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:08:55.341881 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"spinlock_wait_cycles":2816,"thread_start_us":81,"threads_started":1}
I20251025 20:08:55.342136 26879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.338939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:42231 (local address 127.0.0.1:38028)
1025 20:08:55.339166 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.339179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:55.339254 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:55.339653 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:55.339655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:55.339671 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:55.339913 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.339919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.340865 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.340869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:55.341793 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:55.341802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.341902 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.341920 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:55.341966 (+ 46us) client_negotiation.cc:240] Negotiation successful
1025 20:08:55.342019 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20251025 20:08:55.342422 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.338102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.18.82.65:36767)
1025 20:08:55.338446 (+ 344us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:55.338453 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:55.338466 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:55.338519 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:55.338523 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:55.338584 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:55.338687 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:55.339058 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.339765 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.340815 (+ 1050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.341294 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.341338 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:55.341400 (+ 62us) server_negotiation.cc:300] Negotiation successful
1025 20:08:55.341463 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20251025 20:08:55.342504 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.338477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.18.82.67:38011)
1025 20:08:55.339512 (+ 1035us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:55.339517 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:55.339531 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:08:55.339577 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:55.339580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:55.339628 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:55.339716 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:55.340071 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.340544 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.341705 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.342190 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.342230 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:55.342304 (+ 74us) server_negotiation.cc:300] Negotiation successful
1025 20:08:55.342377 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":69,"threads_started":1}
I20251025 20:08:55.342941 26441 tablet_service.cc:1505] Processing CreateTablet for tablet 7231e58d87f14bde9874b624ba9ba382 (DEFAULT_TABLE table=test-table [id=1deb8eb1466146c3af558e3a97c7ae15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:55.343004 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.339071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.18.82.66:42231)
1025 20:08:55.339331 (+ 260us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:55.339335 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:55.339347 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:55.339398 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:55.339401 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:55.339456 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:55.339530 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:55.340036 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.340745 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.342216 (+ 1471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.342729 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.342767 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:55.342814 (+ 47us) server_negotiation.cc:300] Negotiation successful
1025 20:08:55.342875 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20251025 20:08:55.343078 26712 tablet_service.cc:1505] Processing CreateTablet for tablet 7231e58d87f14bde9874b624ba9ba382 (DEFAULT_TABLE table=test-table [id=1deb8eb1466146c3af558e3a97c7ae15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:55.343253 26441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7231e58d87f14bde9874b624ba9ba382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:55.343364 26712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7231e58d87f14bde9874b624ba9ba382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:55.343510 26577 tablet_service.cc:1505] Processing CreateTablet for tablet 7231e58d87f14bde9874b624ba9ba382 (DEFAULT_TABLE table=test-table [id=1deb8eb1466146c3af558e3a97c7ae15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:55.343796 26577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7231e58d87f14bde9874b624ba9ba382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:55.345759 26881 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Bootstrap starting.
I20251025 20:08:55.346231 26883 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Bootstrap starting.
I20251025 20:08:55.346452 26882 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Bootstrap starting.
I20251025 20:08:55.346719 26881 tablet_bootstrap.cc:654] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:55.347258 26882 tablet_bootstrap.cc:654] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:55.347337 26883 tablet_bootstrap.cc:654] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:55.348021 26881 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: No bootstrap required, opened a new log
I20251025 20:08:55.348089 26881 ts_tablet_manager.cc:1403] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:55.348559 26883 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: No bootstrap required, opened a new log
I20251025 20:08:55.348546 26881 raft_consensus.cc:359] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.348647 26883 ts_tablet_manager.cc:1403] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:55.348819 26881 raft_consensus.cc:385] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:55.348920 26881 raft_consensus.cc:740] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14185135b8d443c2aeecd6d5717b930f, State: Initialized, Role: FOLLOWER
I20251025 20:08:55.349109 26881 consensus_queue.cc:260] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.349284 26883 raft_consensus.cc:359] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.349525 26883 raft_consensus.cc:385] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:55.349610 26883 raft_consensus.cc:740] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b446012218f4316a16864099e5bdbaf, State: Initialized, Role: FOLLOWER
I20251025 20:08:55.349637 26881 ts_tablet_manager.cc:1434] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:55.349833 26883 consensus_queue.cc:260] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.350306 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.350040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:45003)
1025 20:08:55.350195 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.350244 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20251025 20:08:55.351470 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.351070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:46119)
1025 20:08:55.351363 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.351411 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20251025 20:08:55.352494 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.352259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:44307)
1025 20:08:55.352389 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.352430 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251025 20:08:55.353417 26781 heartbeater.cc:499] Master 127.18.82.125:45489 was elected leader, sending a full tablet report...
I20251025 20:08:55.353453 26883 ts_tablet_manager.cc:1434] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 20:08:55.354406 26882 tablet_bootstrap.cc:492] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: No bootstrap required, opened a new log
I20251025 20:08:55.354543 26882 ts_tablet_manager.cc:1403] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251025 20:08:55.355011 26882 raft_consensus.cc:359] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.355086 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.354819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:47677)
1025 20:08:55.354999 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.355037 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20251025 20:08:55.355173 26882 raft_consensus.cc:385] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:55.355226 26645 heartbeater.cc:499] Master 127.18.82.125:45489 was elected leader, sending a full tablet report...
I20251025 20:08:55.355237 26882 raft_consensus.cc:740] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9719aa0b9f60432783f766daf87752fd, State: Initialized, Role: FOLLOWER
I20251025 20:08:55.355394 26882 consensus_queue.cc:260] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.355724 26882 ts_tablet_manager.cc:1434] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:55.356442 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.356141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:36115)
1025 20:08:55.356295 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.356356 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20251025 20:08:55.356997 26577 tablet_service.cc:1552] Processing DeleteTablet for tablet c341a19e45af4392a17c566d97d9c22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:51 UTC) from {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:38028
I20251025 20:08:55.357700 26889 tablet_replica.cc:333] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: stopping tablet replica
I20251025 20:08:55.358016 26889 raft_consensus.cc:2243] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:55.358086 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.357857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:57245)
1025 20:08:55.357970 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.358023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251025 20:08:55.358304 26509 heartbeater.cc:499] Master 127.18.82.125:45489 was elected leader, sending a full tablet report...
I20251025 20:08:55.358337 26889 raft_consensus.cc:2272] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:55.359483 26889 ts_tablet_manager.cc:1916] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:55.361663 26889 ts_tablet_manager.cc:1929] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:55.361737 26889 log.cc:1199] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/c341a19e45af4392a17c566d97d9c22d
I20251025 20:08:55.362007 26889 ts_tablet_manager.cc:1950] T c341a19e45af4392a17c566d97d9c22d P 9b446012218f4316a16864099e5bdbaf: Deleting consensus metadata
I20251025 20:08:55.362977 26114 catalog_manager.cc:4985] TS 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231): tablet c341a19e45af4392a17c566d97d9c22d (table test-table [id=9ac86d2fa7744d21bc4bc118bdc53c67]) successfully deleted
I20251025 20:08:55.417898 26888 raft_consensus.cc:493] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:55.418026 26888 raft_consensus.cc:515] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.418401 26888 leader_election.cc:290] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:55.418874 26597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7231e58d87f14bde9874b624ba9ba382" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b446012218f4316a16864099e5bdbaf" is_pre_election: true
I20251025 20:08:55.418892 26733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7231e58d87f14bde9874b624ba9ba382" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14185135b8d443c2aeecd6d5717b930f" is_pre_election: true
I20251025 20:08:55.419020 26597 raft_consensus.cc:2468] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 0.
I20251025 20:08:55.419044 26733 raft_consensus.cc:2468] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 0.
I20251025 20:08:55.419384 26395 leader_election.cc:304] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9719aa0b9f60432783f766daf87752fd, 9b446012218f4316a16864099e5bdbaf; no voters:
I20251025 20:08:55.419552 26888 raft_consensus.cc:2804] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:55.419611 26888 raft_consensus.cc:493] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:55.419672 26888 raft_consensus.cc:3060] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:55.420555 26888 raft_consensus.cc:515] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.420869 26888 leader_election.cc:290] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 election: Requested vote from peers 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011), 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231)
I20251025 20:08:55.421252 26597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7231e58d87f14bde9874b624ba9ba382" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b446012218f4316a16864099e5bdbaf"
I20251025 20:08:55.421295 26733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7231e58d87f14bde9874b624ba9ba382" candidate_uuid: "9719aa0b9f60432783f766daf87752fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14185135b8d443c2aeecd6d5717b930f"
I20251025 20:08:55.421394 26597 raft_consensus.cc:3060] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:55.421479 26733 raft_consensus.cc:3060] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:55.422235 26597 raft_consensus.cc:2468] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 1.
I20251025 20:08:55.422369 26733 raft_consensus.cc:2468] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719aa0b9f60432783f766daf87752fd in term 1.
I20251025 20:08:55.422520 26395 leader_election.cc:304] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9719aa0b9f60432783f766daf87752fd, 9b446012218f4316a16864099e5bdbaf; no voters:
I20251025 20:08:55.422698 26888 raft_consensus.cc:2804] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:55.422904 26888 raft_consensus.cc:697] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 1 LEADER]: Becoming Leader. State: Replica: 9719aa0b9f60432783f766daf87752fd, State: Running, Role: LEADER
I20251025 20:08:55.423203 26888 consensus_queue.cc:237] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } }
I20251025 20:08:55.423602 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.423327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:57569)
1025 20:08:55.423483 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.423526 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20251025 20:08:55.425058 26121 catalog_manager.cc:5649] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9719aa0b9f60432783f766daf87752fd (127.18.82.65). New cstate: current_term: 1 leader_uuid: "9719aa0b9f60432783f766daf87752fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9719aa0b9f60432783f766daf87752fd" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:55.439188 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.436762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.18.82.65:36767)
1025 20:08:55.436895 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:55.436902 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:55.436918 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:08:55.436966 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:55.436968 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:55.437009 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:55.437072 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:55.437397 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.437976 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:55.438601 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:55.438765 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:55.438803 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:55.438864 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:55.438925 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:55.439030 (+ 105us) server_negotiation.cc:300] Negotiation successful
1025 20:08:55.439072 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 20:08:55.440929 26597 raft_consensus.cc:1275] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Refusing update from remote peer 9719aa0b9f60432783f766daf87752fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:55.441035 26733 raft_consensus.cc:1275] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Refusing update from remote peer 9719aa0b9f60432783f766daf87752fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:55.441500 26888 consensus_queue.cc:1048] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b446012218f4316a16864099e5bdbaf" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:55.441701 26890 consensus_queue.cc:1048] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14185135b8d443c2aeecd6d5717b930f" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:55.443190 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.442915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:35661)
1025 20:08:55.443031 (+ 116us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.443100 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251025 20:08:55.443517 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.443130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:42163)
1025 20:08:55.443386 (+ 256us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.443444 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20251025 20:08:55.446663 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.446394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.65:39451)
1025 20:08:55.446537 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.446589 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251025 20:08:55.447213 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.446186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.67:57835)
1025 20:08:55.447089 (+ 903us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.447141 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834}
I20251025 20:08:55.447649 26893 mvcc.cc:204] Tried to move back new op lower bound from 7214788343562805248 to 7214788343494909952. Current Snapshot: MvccSnapshot[applied={T|T < 7214788343562805248 or (T in {7214788343562805248})}]
I20251025 20:08:55.448482 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:55.448230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:45153 (local address 127.18.82.66:56041)
1025 20:08:55.448372 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:55.448417 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20251025 20:08:55.450933 26129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42794:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:55.451071 26129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42794:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:55.454365 26129 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 83e87b1cb0ca4b91aec6bc3ff3426fe4: Sending DeleteTablet for 3 replicas of tablet 7231e58d87f14bde9874b624ba9ba382
I20251025 20:08:55.454941 26712 tablet_service.cc:1552] Processing DeleteTablet for tablet 7231e58d87f14bde9874b624ba9ba382 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:55 UTC) from {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:40996
I20251025 20:08:55.454970 26441 tablet_service.cc:1552] Processing DeleteTablet for tablet 7231e58d87f14bde9874b624ba9ba382 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:55 UTC) from {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:43634
I20251025 20:08:55.455061 26577 tablet_service.cc:1552] Processing DeleteTablet for tablet 7231e58d87f14bde9874b624ba9ba382 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:55 UTC) from {username='oryx', principal='oryx/127.18.82.125@KRBTEST.COM'} at 127.0.0.1:38028
I20251025 20:08:55.455274 26889 tablet_replica.cc:333] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: stopping tablet replica
I20251025 20:08:55.455329 26902 tablet_replica.cc:333] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: stopping tablet replica
I20251025 20:08:55.455420 26889 raft_consensus.cc:2243] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:55.455447 26903 tablet_replica.cc:333] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: stopping tablet replica
I20251025 20:08:55.455439 26902 raft_consensus.cc:2243] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:55.455535 26889 raft_consensus.cc:2272] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:55.455551 26902 raft_consensus.cc:2272] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:55.455529 26903 raft_consensus.cc:2243] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:55.455701 26903 raft_consensus.cc:2272] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:55.456233 26902 ts_tablet_manager.cc:1916] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:55.456262 26903 ts_tablet_manager.cc:1916] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:55.456262 26889 ts_tablet_manager.cc:1916] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:55.458091 26902 ts_tablet_manager.cc:1929] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:55.458160 26902 log.cc:1199] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/7231e58d87f14bde9874b624ba9ba382
I20251025 20:08:55.458423 26889 ts_tablet_manager.cc:1929] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:55.458423 26903 ts_tablet_manager.cc:1929] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:55.458457 26902 ts_tablet_manager.cc:1950] T 7231e58d87f14bde9874b624ba9ba382 P 14185135b8d443c2aeecd6d5717b930f: Deleting consensus metadata
I20251025 20:08:55.458499 26889 log.cc:1199] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/7231e58d87f14bde9874b624ba9ba382
I20251025 20:08:55.458499 26903 log.cc:1199] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/7231e58d87f14bde9874b624ba9ba382
I20251025 20:08:55.458755 26889 ts_tablet_manager.cc:1950] T 7231e58d87f14bde9874b624ba9ba382 P 9b446012218f4316a16864099e5bdbaf: Deleting consensus metadata
I20251025 20:08:55.458786 26903 ts_tablet_manager.cc:1950] T 7231e58d87f14bde9874b624ba9ba382 P 9719aa0b9f60432783f766daf87752fd: Deleting consensus metadata
I20251025 20:08:55.459071 26115 catalog_manager.cc:4985] TS 14185135b8d443c2aeecd6d5717b930f (127.18.82.67:38011): tablet 7231e58d87f14bde9874b624ba9ba382 (table test-table [id=1deb8eb1466146c3af558e3a97c7ae15]) successfully deleted
I20251025 20:08:55.459345 26114 catalog_manager.cc:4985] TS 9b446012218f4316a16864099e5bdbaf (127.18.82.66:42231): tablet 7231e58d87f14bde9874b624ba9ba382 (table test-table [id=1deb8eb1466146c3af558e3a97c7ae15]) successfully deleted
I20251025 20:08:55.459458 26113 catalog_manager.cc:4985] TS 9719aa0b9f60432783f766daf87752fd (127.18.82.65:36767): tablet 7231e58d87f14bde9874b624ba9ba382 (table test-table [id=1deb8eb1466146c3af558e3a97c7ae15]) successfully deleted
I20251025 20:08:55.459635 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 26378
I20251025 20:08:55.468060 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 26514
I20251025 20:08:55.473889 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 26650
I20251025 20:08:55.479759 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 25983
I20251025 20:08:55.485769 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 26101
2025-10-25T20:08:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9235 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:55 dist-test-slave-d2pk krb5kdc[26906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:55 dist-test-slave-d2pk krb5kdc[26906](info): set up 2 sockets
Oct 25 20:08:55 dist-test-slave-d2pk krb5kdc[26906](info): commencing operation
krb5kdc: starting...
W20251025 20:08:57.434286 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.925s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:08:57 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:08:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:08:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:57.560596 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:40179
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:57.664489 26922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:57.664707 26922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:57.664762 26922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:57.667843 26922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:08:57.667892 26922 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 20:08:57.667909 26922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:57.667928 26922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:57.667951 26922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:08:57.667966 26922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:08:57.671990 26922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:40179
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:57.672979 26922 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:57.673810 26922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:57.679400 26928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:57.679476 26930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:57.679561 26922 server_base.cc:1047] running on GCE node
W20251025 20:08:57.679476 26927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:57.680125 26922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:57.680927 26922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:57.682066 26922 hybrid_clock.cc:648] HybridClock initialized: now 1761422937682056 us; error 24 us; skew 500 ppm
Oct 25 20:08:57 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:57.684410 26922 init.cc:377] Logged in from keytab as oryx/127.18.82.126@KRBTEST.COM (short username oryx)
I20251025 20:08:57.685400 26922 webserver.cc:492] Webserver started at http://127.18.82.126:43643/ using document root <none> and password file <none>
I20251025 20:08:57.685935 26922 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:57.685983 26922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:57.686163 26922 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:57.687856 26922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "3e9112ed74d6496b8c872a3f16732d90"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "3dfd9b2831341434af28407d82c8425c"
server_key_iv: "89e755dcaa4124f2dc7a766a852fd7cd"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.688292 26922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "3e9112ed74d6496b8c872a3f16732d90"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "3dfd9b2831341434af28407d82c8425c"
server_key_iv: "89e755dcaa4124f2dc7a766a852fd7cd"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.691501 26922 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:08:57.693661 26937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:57.694648 26922 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:57.694777 26922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "3e9112ed74d6496b8c872a3f16732d90"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "3dfd9b2831341434af28407d82c8425c"
server_key_iv: "89e755dcaa4124f2dc7a766a852fd7cd"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.694861 26922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:57.710726 26922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:57.718668 26922 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:57.718835 26922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:57.726490 26922 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:40179
I20251025 20:08:57.726749 26999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:40179 every 8 connection(s)
I20251025 20:08:57.727466 26922 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:08:57.730218 27000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:57.735037 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Bootstrap starting.
I20251025 20:08:57.735729 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 26922
I20251025 20:08:57.735841 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:08:57.736034 18761 external_mini_cluster.cc:1468] Setting key 17d7b1021b1e3e1e85026a57a8e26876
I20251025 20:08:57.737473 27000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:57.738334 27000 log.cc:826] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:57.739990 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: No bootstrap required, opened a new log
I20251025 20:08:57.743161 27000 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } }
I20251025 20:08:57.743415 27000 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:57.743503 27000 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9112ed74d6496b8c872a3f16732d90, State: Initialized, Role: FOLLOWER
Oct 25 20:08:57 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:57.743939 27000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } }
I20251025 20:08:57.744087 27000 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:08:57.744153 27000 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:08:57.744227 27000 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:57.745877 27000 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } }
I20251025 20:08:57.746238 27000 leader_election.cc:304] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9112ed74d6496b8c872a3f16732d90; no voters:
I20251025 20:08:57.746613 27000 leader_election.cc:290] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:08:57.746704 27005 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:57.746917 27005 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9112ed74d6496b8c872a3f16732d90, State: Running, Role: LEADER
I20251025 20:08:57.747251 27005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } }
I20251025 20:08:57.747815 27000 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:08:57.748876 27007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9112ed74d6496b8c872a3f16732d90. Latest consensus state: current_term: 1 leader_uuid: "3e9112ed74d6496b8c872a3f16732d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } } }
I20251025 20:08:57.748890 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9112ed74d6496b8c872a3f16732d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9112ed74d6496b8c872a3f16732d90" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 40179 } } }
I20251025 20:08:57.749078 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:57.749377 27007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90 [sys.catalog]: This master's current role is: LEADER
I20251025 20:08:57.749480 27014 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:08:57.750097 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:57.737059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.18.82.126:40179)
1025 20:08:57.737529 (+ 470us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:57.737549 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:57.737587 (+ 38us) server_negotiation.cc:408] Connection header received
1025 20:08:57.738272 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:57.738297 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:57.738667 (+ 370us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:57.738981 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:57.739826 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:57.740738 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:57.741640 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:57.741926 (+ 286us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:57.744533 (+ 2607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:57.744563 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:57.744592 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:57.744640 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:57.746752 (+ 2112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:57.747235 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:57.747240 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:57.747245 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:57.747370 (+ 125us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:57.747634 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:57.747637 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:57.747639 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:57.748065 (+ 426us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:57.748265 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:57.749036 (+ 771us) server_negotiation.cc:300] Negotiation successful
1025 20:08:57.749377 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20251025 20:08:57.752565 27014 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:08:57.757568 27014 catalog_manager.cc:1357] Generated new cluster ID: fecbec47bdf644fa9ee3306154e15157
I20251025 20:08:57.757628 27014 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:08:57.775827 27014 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:08:57.776714 27014 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:08:57.790108 27014 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Generated new TSK 0
I20251025 20:08:57.790863 27014 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:57.841990 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:40179
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:57.949497 27028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:57.949718 27028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:57.949779 27028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:57.952739 27028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:57.952787 27028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:57.952854 27028 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:57.952893 27028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:08:57.956759 27028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:57.957775 27028 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:57.958601 27028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:57.964722 27033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:57.965023 27028 server_base.cc:1047] running on GCE node
W20251025 20:08:57.965052 27034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:57.964740 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:57.965497 27028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:57.966020 27028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:57.967162 27028 hybrid_clock.cc:648] HybridClock initialized: now 1761422937967115 us; error 63 us; skew 500 ppm
Oct 25 20:08:57 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:57.969882 27028 init.cc:377] Logged in from keytab as oryx/127.18.82.65@KRBTEST.COM (short username oryx)
I20251025 20:08:57.970955 27028 webserver.cc:492] Webserver started at http://127.18.82.65:40935/ using document root <none> and password file <none>
I20251025 20:08:57.971454 27028 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:57.971503 27028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:57.971678 27028 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:57.973286 27028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "648cb14e60c327e7861fceac33b39b75"
server_key_iv: "20e2c32d5163220626354a52fd059c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.973780 27028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "648cb14e60c327e7861fceac33b39b75"
server_key_iv: "20e2c32d5163220626354a52fd059c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.976966 27028 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251025 20:08:57.979090 27043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:57.980049 27028 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 20:08:57.980141 27028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
format_stamp: "Formatted at 2025-10-25 20:08:57 on dist-test-slave-d2pk"
server_key: "648cb14e60c327e7861fceac33b39b75"
server_key_iv: "20e2c32d5163220626354a52fd059c2c"
server_key_version: "encryptionkey@0"
I20251025 20:08:57.980225 27028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:57.993000 27028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:57.995493 27028 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:57.995663 27028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:57.997051 27028 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:57.997097 27028 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:57.997136 27028 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:57.997169 27028 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:57 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:58.007331 27028 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:36159
I20251025 20:08:58.007354 27158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:36159 every 8 connection(s)
I20251025 20:08:58.008404 27028 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:08:58.009852 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:57.997929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.65:45935)
1025 20:08:57.998342 (+ 413us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:57.998381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:57.999240 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:57.999964 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:57.999976 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.000537 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.001256 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.001276 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.002307 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.002313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.002887 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.002900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.003120 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.003780 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.003818 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.005531 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.007508 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.007512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.007520 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.007732 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.007978 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.007980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.007981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.008042 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.008440 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.008445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.008683 (+ 238us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.008890 (+ 207us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.009166 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20251025 20:08:58.009958 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:57.997664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:45935 (local address 127.18.82.126:40179)
1025 20:08:57.997823 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:57.997829 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:57.998484 (+ 655us) server_negotiation.cc:408] Connection header received
1025 20:08:57.999498 (+ 1014us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:57.999504 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:57.999562 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:57.999652 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.001429 (+ 1777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.002173 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.003014 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.003179 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.005700 (+ 2521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.005722 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.005725 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.005753 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.007361 (+ 1608us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.007827 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.007831 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.007833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.007886 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.008118 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.008124 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.008126 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.008332 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.008460 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.009683 (+ 1223us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.009832 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:08:58.010929 27159 heartbeater.cc:344] Connected to a master server at 127.18.82.126:40179
I20251025 20:08:58.011073 27159 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:58.011520 27159 heartbeater.cc:507] Master 127.18.82.126:40179 requested a full tablet report, sending...
I20251025 20:08:58.012975 26954 ts_manager.cc:194] Registered new tserver with Master: ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159)
I20251025 20:08:58.014386 26954 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:45935
I20251025 20:08:58.017767 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27028
I20251025 20:08:58.017891 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:08:58.018131 18761 external_mini_cluster.cc:1468] Setting key 4ea69b644ae90dcdac35e4861999b15f
I20251025 20:08:58.018754 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.012883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:41711 (local address 127.18.82.126:40179)
1025 20:08:58.012982 (+ 99us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.012985 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.013201 (+ 216us) server_negotiation.cc:408] Connection header received
1025 20:08:58.013317 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.013319 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.013359 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.013423 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.014047 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.014551 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.015180 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.015321 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.016113 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.016126 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.016129 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.016149 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.017362 (+ 1213us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.017740 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.017744 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.017747 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.017800 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.018065 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.018068 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.018069 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.018188 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.018273 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.018532 (+ 259us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.018656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251025 20:08:58.018759 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.012829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.65:41711)
1025 20:08:58.013125 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.013140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.013234 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.013453 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.013456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.013672 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.013938 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.013947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.014651 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.014655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.015072 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.015084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.015193 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.015673 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.015688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.016007 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.017452 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.017454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.017456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.017646 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.017904 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.017907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.017909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.017970 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.018292 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.018296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.018382 (+ 86us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.018490 (+ 108us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.018601 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:58.060207 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:40179
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:58.163426 27170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:58.163641 27170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:58.163733 27170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:58.166905 27170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:58.166975 27170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:58.167058 27170 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:58.167132 27170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:08:58.171057 27170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:58.172050 27170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:58.172864 27170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:58.179162 27175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:58.179219 27176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:58.179142 27178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:58.179364 27170 server_base.cc:1047] running on GCE node
I20251025 20:08:58.179734 27170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:58.180276 27170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:58.181435 27170 hybrid_clock.cc:648] HybridClock initialized: now 1761422938181414 us; error 32 us; skew 500 ppm
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:58.184016 27170 init.cc:377] Logged in from keytab as oryx/127.18.82.66@KRBTEST.COM (short username oryx)
I20251025 20:08:58.185055 27170 webserver.cc:492] Webserver started at http://127.18.82.66:45749/ using document root <none> and password file <none>
I20251025 20:08:58.185618 27170 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:58.185672 27170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:58.185830 27170 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:58.187444 27170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "a03af3799333416fb8640f01f8ee755f"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "6578e23b8646f3ab689ad4e2b87d4e98"
server_key_iv: "2c42c9bf7704bf1aea3c7dff8cb17329"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.187876 27170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "a03af3799333416fb8640f01f8ee755f"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "6578e23b8646f3ab689ad4e2b87d4e98"
server_key_iv: "2c42c9bf7704bf1aea3c7dff8cb17329"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.191038 27170 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:08:58.192999 27185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:58.194053 27170 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:58.194164 27170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "a03af3799333416fb8640f01f8ee755f"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "6578e23b8646f3ab689ad4e2b87d4e98"
server_key_iv: "2c42c9bf7704bf1aea3c7dff8cb17329"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.194243 27170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:58.205965 27170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:58.208307 27170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:58.208467 27170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:58.209825 27170 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:58.209873 27170 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:58.209928 27170 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:58.209956 27170 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:58.221318 27170 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:37991
I20251025 20:08:58.221329 27300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:37991 every 8 connection(s)
I20251025 20:08:58.221639 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.210357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:35317 (local address 127.18.82.126:40179)
1025 20:08:58.210508 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.210513 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.211222 (+ 709us) server_negotiation.cc:408] Connection header received
1025 20:08:58.212350 (+ 1128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.212353 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.212408 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.212508 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.214265 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.214891 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.215532 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.215715 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.218252 (+ 2537us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.218277 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.218280 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.218314 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.219842 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.220314 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.220317 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.220318 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.220368 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.220628 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.220632 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.220634 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.220774 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.220864 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.221326 (+ 462us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.221446 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 20:08:58.222179 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.210602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.66:35317)
1025 20:08:58.211056 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.211104 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.212121 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.212706 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.212720 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.213283 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.214109 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.214127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.215019 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.215023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.215420 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.215428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.215632 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.216208 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.216246 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.218109 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.219970 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.219976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.219987 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.220215 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.220451 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.220454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.220456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.220533 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.220875 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.220883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.221105 (+ 222us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.221278 (+ 173us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.221546 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20251025 20:08:58.222380 27170 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:08:58.223901 27301 heartbeater.cc:344] Connected to a master server at 127.18.82.126:40179
I20251025 20:08:58.224046 27301 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:58.224535 27301 heartbeater.cc:507] Master 127.18.82.126:40179 requested a full tablet report, sending...
I20251025 20:08:58.224967 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27170
I20251025 20:08:58.225059 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:08:58.225370 18761 external_mini_cluster.cc:1468] Setting key 4f52c811ac6cd98142b0fec8925764b2
I20251025 20:08:58.225440 26953 ts_manager.cc:194] Registered new tserver with Master: a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991)
I20251025 20:08:58.226112 26953 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:35317
I20251025 20:08:58.231638 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.225391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:41763 (local address 127.18.82.126:40179)
1025 20:08:58.225499 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.225503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.225789 (+ 286us) server_negotiation.cc:408] Connection header received
1025 20:08:58.226054 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.226056 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.226090 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.226157 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.226793 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.227238 (+ 445us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.227886 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.228090 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.228978 (+ 888us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.228991 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.228994 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.229015 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.230319 (+ 1304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.230680 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.230684 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.230685 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.230723 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.230942 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.230945 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.230947 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.231091 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.231177 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.231407 (+ 230us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.231508 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 20:08:58.231657 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.225352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.66:41763)
1025 20:08:58.225707 (+ 355us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.225721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.225853 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.226197 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.226200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.226405 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.226690 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.226700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.227338 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.227342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.227781 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.227788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.227893 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.228441 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.228461 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.228858 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.230433 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.230437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.230439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.230600 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.230799 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.230802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.230803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.230843 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.231178 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.231181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.231255 (+ 74us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.231341 (+ 86us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.231466 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab.
I20251025 20:08:58.269868 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:40179
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:08:58.372787 27312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:08:58.372972 27312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:08:58.373026 27312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:08:58.376055 27312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:08:58.376104 27312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:08:58.376147 27312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 20:08:58.376185 27312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:08:58.379782 27312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.82.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:08:58.380683 27312 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:08:58.381482 27312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:08:58.387712 27318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:08:58.387745 27320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:58.387813 27312 server_base.cc:1047] running on GCE node
W20251025 20:08:58.387718 27317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:08:58.388245 27312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:08:58.388747 27312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:08:58.389881 27312 hybrid_clock.cc:648] HybridClock initialized: now 1761422938389869 us; error 28 us; skew 500 ppm
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:08:58.392246 27312 init.cc:377] Logged in from keytab as oryx/127.18.82.67@KRBTEST.COM (short username oryx)
I20251025 20:08:58.393242 27312 webserver.cc:492] Webserver started at http://127.18.82.67:32775/ using document root <none> and password file <none>
I20251025 20:08:58.393821 27312 fs_manager.cc:362] Metadata directory not provided
I20251025 20:08:58.393872 27312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:08:58.394037 27312 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:08:58.395617 27312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "4c194dacee924f0b8e081f986bf09cbe"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "458395eef2d4b28b2761f64a97c39061"
server_key_iv: "89f992dbec11ba659117ba5813299f8a"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.396075 27312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "4c194dacee924f0b8e081f986bf09cbe"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "458395eef2d4b28b2761f64a97c39061"
server_key_iv: "89f992dbec11ba659117ba5813299f8a"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.399291 27312 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:08:58.401263 27327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:58.402232 27312 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:08:58.402320 27312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "4c194dacee924f0b8e081f986bf09cbe"
format_stamp: "Formatted at 2025-10-25 20:08:58 on dist-test-slave-d2pk"
server_key: "458395eef2d4b28b2761f64a97c39061"
server_key_iv: "89f992dbec11ba659117ba5813299f8a"
server_key_version: "encryptionkey@0"
I20251025 20:08:58.402410 27312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:08:58.430605 27312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:08:58.433204 27312 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:08:58.433373 27312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:08:58.434906 27312 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:08:58.434957 27312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:08:58.435017 27312 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:08:58.435045 27312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:58.445559 27312 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:45361
I20251025 20:08:58.445581 27442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:45361 every 8 connection(s)
I20251025 20:08:58.446655 27312 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:08:58.447921 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.435793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.67:35835)
1025 20:08:58.436502 (+ 709us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.436572 (+ 70us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.437671 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.438213 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.438225 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.438726 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.439364 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.439378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.440419 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.440423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.441045 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.441055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.441285 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.441923 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.441954 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.443809 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.445806 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.445810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.445819 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.446027 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.446249 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.446250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.446252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.446329 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.446715 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.446723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.446931 (+ 208us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.447114 (+ 183us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.447343 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":127,"threads_started":1}
I20251025 20:08:58.448458 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.435533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:35835 (local address 127.18.82.126:40179)
1025 20:08:58.435670 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.435675 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.436727 (+ 1052us) server_negotiation.cc:408] Connection header received
1025 20:08:58.437899 (+ 1172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.437903 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.437957 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.438045 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.439516 (+ 1471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.440288 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.441191 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.441416 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.443980 (+ 2564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.444003 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.444006 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.444040 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.445698 (+ 1658us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.446114 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.446118 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.446120 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.446170 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.446419 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.446423 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.446425 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.446610 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.446719 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.448261 (+ 1542us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.448354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 20:08:58.449333 27443 heartbeater.cc:344] Connected to a master server at 127.18.82.126:40179
I20251025 20:08:58.449496 27443 heartbeater.cc:461] Registering TS with master...
I20251025 20:08:58.449981 27443 heartbeater.cc:507] Master 127.18.82.126:40179 requested a full tablet report, sending...
I20251025 20:08:58.450901 26953 ts_manager.cc:194] Registered new tserver with Master: 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:58.451534 26953 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:35835
I20251025 20:08:58.454854 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27312
I20251025 20:08:58.454962 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:08:58.455232 18761 external_mini_cluster.cc:1468] Setting key 6fa9bfc4d8fe98a10d4bdc60bde9ba4b
I20251025 20:08:58.456912 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.451129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.18.82.67:50685)
1025 20:08:58.451440 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.451453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.451546 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.451784 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.451786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.451970 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.452182 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.452188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.452868 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.452872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.453216 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.453223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.453320 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.453845 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.453857 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.454108 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.455754 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.455757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.455759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.455957 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.456156 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.456159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.456161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.456209 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.456518 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.456520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.456596 (+ 76us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.456681 (+ 85us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.456771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20251025 20:08:58.456980 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.451194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:50685 (local address 127.18.82.126:40179)
1025 20:08:58.451300 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.451304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.451518 (+ 214us) server_negotiation.cc:408] Connection header received
1025 20:08:58.451641 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.451646 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.451689 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.451749 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.452289 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.452753 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.453313 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.453566 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.454221 (+ 655us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.454236 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.454238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.454259 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.455651 (+ 1392us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.456038 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.456041 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.456043 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.456082 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.456286 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.456289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.456291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.456428 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.456522 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.456739 (+ 217us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.456840 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251025 20:08:58.457098 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
W20251025 20:08:58.477224 27003 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:58.472365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.18.82.126:40179)
1025 20:08:58.472482 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.472485 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.472497 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:08:58.472535 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.472538 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.472575 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.472644 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.473335 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.473888 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.474497 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.474628 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.477124 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50528: BlockingRecv error: recv got EOF from 127.0.0.1:50528 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:58.484966 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.478496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.18.82.126:40179)
1025 20:08:58.478615 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.478618 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.478705 (+ 87us) server_negotiation.cc:408] Connection header received
1025 20:08:58.478782 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.478785 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.478827 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.478901 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.479569 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.480006 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.480478 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.480582 (+ 104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.482438 (+ 1856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.482453 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.482455 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.482477 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.483648 (+ 1171us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.483986 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.483989 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.483990 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.484030 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.484233 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.484237 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.484238 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.484389 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.484513 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.484727 (+ 214us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.484829 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:08:58.487250 26953 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:58.489589 26953 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:58.501920 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.498171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.0.0.1:39540)
1025 20:08:58.498798 (+ 627us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.498810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.498878 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.499279 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.499282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.499304 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.499545 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.499552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.500644 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.500647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.501473 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.501482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.501612 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.501630 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.501698 (+ 68us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.501775 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":80,"threads_started":1}
I20251025 20:08:58.502068 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.497240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.18.82.65:36159)
1025 20:08:58.497626 (+ 386us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.497634 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.497756 (+ 122us) server_negotiation.cc:408] Connection header received
1025 20:08:58.498068 (+ 312us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.498081 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.498280 (+ 199us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.498420 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.498889 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.499930 (+ 1041us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.501046 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.501391 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.501474 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.501867 (+ 393us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.501955 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
I20251025 20:08:58.502148 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.497821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37991 (local address 127.0.0.1:55510)
1025 20:08:58.498500 (+ 679us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.498513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.498606 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.499496 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.499499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.499515 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.499712 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.499718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.501019 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.501023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.501837 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.501845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.501930 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.501948 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.501994 (+ 46us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.502053 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":104,"threads_started":1}
I20251025 20:08:58.502394 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.497154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36159 (local address 127.0.0.1:37928)
1025 20:08:58.497482 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.497681 (+ 199us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.497816 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.498439 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.498444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.498472 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.498715 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.498726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.500024 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.500028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.500893 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.500905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.501561 (+ 656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.501620 (+ 59us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.501719 (+ 99us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.501781 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20251025 20:08:58.502406 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.498572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.18.82.67:45361)
1025 20:08:58.498904 (+ 332us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.498908 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.498936 (+ 28us) server_negotiation.cc:408] Connection header received
1025 20:08:58.499026 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.499031 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.499144 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.499259 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.499675 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.500516 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.501613 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.502040 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.502129 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.502193 (+ 64us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.502274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20251025 20:08:58.503065 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.498159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.18.82.66:37991)
1025 20:08:58.499121 (+ 962us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.499127 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.499147 (+ 20us) server_negotiation.cc:408] Connection header received
1025 20:08:58.499221 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.499226 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.499366 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.499498 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.499816 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.500842 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.502240 (+ 1398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.502765 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.502858 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.502905 (+ 47us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.502966 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":88,"threads_started":1}
I20251025 20:08:58.505030 27377 tablet_service.cc:1505] Processing CreateTablet for tablet a2a7bbada61b415c868dc235ca7c0406 (DEFAULT_TABLE table=test-table [id=0fe7d54bd0264278be993c5b2841dc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.505936 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a7bbada61b415c868dc235ca7c0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.507347 27235 tablet_service.cc:1505] Processing CreateTablet for tablet a2a7bbada61b415c868dc235ca7c0406 (DEFAULT_TABLE table=test-table [id=0fe7d54bd0264278be993c5b2841dc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.507355 27093 tablet_service.cc:1505] Processing CreateTablet for tablet a2a7bbada61b415c868dc235ca7c0406 (DEFAULT_TABLE table=test-table [id=0fe7d54bd0264278be993c5b2841dc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.508240 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a7bbada61b415c868dc235ca7c0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.508240 27235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a7bbada61b415c868dc235ca7c0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.511570 27471 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Bootstrap starting.
I20251025 20:08:58.513628 27471 tablet_bootstrap.cc:654] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.514072 27472 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Bootstrap starting.
I20251025 20:08:58.514468 27473 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Bootstrap starting.
I20251025 20:08:58.515564 27471 log.cc:826] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:58.516446 27472 tablet_bootstrap.cc:654] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.516527 27473 tablet_bootstrap.cc:654] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.517082 27472 log.cc:826] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:58.517560 27471 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: No bootstrap required, opened a new log
I20251025 20:08:58.517706 27473 log.cc:826] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Log is configured to *not* fsync() on all Append() calls
I20251025 20:08:58.517772 27471 ts_tablet_manager.cc:1403] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251025 20:08:58.518682 27472 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: No bootstrap required, opened a new log
I20251025 20:08:58.518839 27472 ts_tablet_manager.cc:1403] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:08:58.519400 27473 tablet_bootstrap.cc:492] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: No bootstrap required, opened a new log
I20251025 20:08:58.519567 27473 ts_tablet_manager.cc:1403] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:08:58.520773 27471 raft_consensus.cc:359] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.520990 27471 raft_consensus.cc:385] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.521090 27471 raft_consensus.cc:740] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.521576 27471 consensus_queue.cc:260] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.522048 27472 raft_consensus.cc:359] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.522239 27443 heartbeater.cc:499] Master 127.18.82.126:40179 was elected leader, sending a full tablet report...
I20251025 20:08:58.522286 27472 raft_consensus.cc:385] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.522336 27472 raft_consensus.cc:740] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.522413 27471 ts_tablet_manager.cc:1434] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:08:58.522548 27473 raft_consensus.cc:359] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.522707 27473 raft_consensus.cc:385] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.522747 27473 raft_consensus.cc:740] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.522709 27472 consensus_queue.cc:260] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.523031 27473 consensus_queue.cc:260] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.523418 27472 ts_tablet_manager.cc:1434] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 20:08:58.523761 27473 ts_tablet_manager.cc:1434] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 20:08:58.523813 27159 heartbeater.cc:499] Master 127.18.82.126:40179 was elected leader, sending a full tablet report...
I20251025 20:08:58.523785 27301 heartbeater.cc:499] Master 127.18.82.126:40179 was elected leader, sending a full tablet report...
I20251025 20:08:58.533998 27477 raft_consensus.cc:493] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:58.534117 27477 raft_consensus.cc:515] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.535027 27477 leader_election.cc:290] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991)
I20251025 20:08:58.537585 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.535221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36159 (local address 127.18.82.67:57871)
1025 20:08:58.535354 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.535371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.535444 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.535794 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.535796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.535812 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.535987 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.535991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.536658 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.536661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.537194 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.537201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.537302 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.537326 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.537386 (+ 60us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.537444 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251025 20:08:58.538023 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.535284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:57871 (local address 127.18.82.65:36159)
1025 20:08:58.535395 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.535399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.535414 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:58.535572 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.535576 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.535623 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.535685 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.536090 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.536537 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.537301 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.537783 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.537824 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.537873 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.537939 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:08:58.538457 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a7bbada61b415c868dc235ca7c0406" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" is_pre_election: true
I20251025 20:08:58.538550 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.535443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37991 (local address 127.18.82.67:33735)
1025 20:08:58.535782 (+ 339us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.535792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.535882 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.536081 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.536084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.536098 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.536339 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.536352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.537024 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.537027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.537638 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.537651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.538284 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.538297 (+ 13us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.538376 (+ 79us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.538429 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20251025 20:08:58.538563 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.535569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:33735 (local address 127.18.82.66:37991)
1025 20:08:58.535710 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.535714 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.535850 (+ 136us) server_negotiation.cc:408] Connection header received
1025 20:08:58.535975 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.535977 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.536012 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.536067 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.536441 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.536921 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.537785 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.538194 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.538230 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.538407 (+ 177us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.538470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251025 20:08:58.538694 27113 raft_consensus.cc:2468] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 0.
I20251025 20:08:58.539105 27330 leader_election.cc:304] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c194dacee924f0b8e081f986bf09cbe, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:58.539163 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a7bbada61b415c868dc235ca7c0406" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f" is_pre_election: true
I20251025 20:08:58.539333 27477 raft_consensus.cc:2804] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:58.539398 27477 raft_consensus.cc:493] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:58.539381 27255 raft_consensus.cc:2468] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 0.
I20251025 20:08:58.539428 27477 raft_consensus.cc:3060] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.540387 27477 raft_consensus.cc:515] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.540695 27477 leader_election.cc:290] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 election: Requested vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991)
I20251025 20:08:58.541013 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a7bbada61b415c868dc235ca7c0406" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f"
I20251025 20:08:58.541018 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a7bbada61b415c868dc235ca7c0406" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
I20251025 20:08:58.541101 27113 raft_consensus.cc:3060] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.541110 27255 raft_consensus.cc:3060] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.542013 27113 raft_consensus.cc:2468] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 1.
I20251025 20:08:58.542069 27255 raft_consensus.cc:2468] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 1.
I20251025 20:08:58.542265 27330 leader_election.cc:304] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c194dacee924f0b8e081f986bf09cbe, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:58.542434 27477 raft_consensus.cc:2804] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:58.542622 27477 raft_consensus.cc:697] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 1 LEADER]: Becoming Leader. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Running, Role: LEADER
I20251025 20:08:58.542905 27477 consensus_queue.cc:237] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.545800 26953 catalog_manager.cc:5649] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67). New cstate: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } health_report { overall_health: HEALTHY } } }
I20251025 20:08:58.565237 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.562282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.18.82.67:45361)
1025 20:08:58.562427 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.562432 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.562447 (+ 15us) server_negotiation.cc:408] Connection header received
1025 20:08:58.562539 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.562542 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.562591 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.562674 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:58.563002 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.563512 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.564137 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.564320 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.564400 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:58.564878 (+ 478us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:58.564979 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.565050 (+ 71us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.565095 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:08:58.570667 27255 raft_consensus.cc:1275] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Refusing update from remote peer 4c194dacee924f0b8e081f986bf09cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:58.570672 27113 raft_consensus.cc:1275] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Refusing update from remote peer 4c194dacee924f0b8e081f986bf09cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:58.571256 27481 consensus_queue.cc:1048] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:58.571425 27477 consensus_queue.cc:1048] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:58.579550 27486 mvcc.cc:204] Tried to move back new op lower bound from 7214788356378505216 to 7214788356274450432. Current Snapshot: MvccSnapshot[applied={T|T < 7214788356378505216 or (T in {7214788356378505216})}]
I20251025 20:08:58.586570 26952 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:58.586726 26952 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:58.589075 26952 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Sending DeleteTablet for 3 replicas of tablet a2a7bbada61b415c868dc235ca7c0406
I20251025 20:08:58.589802 27093 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a7bbada61b415c868dc235ca7c0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:58 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37928
I20251025 20:08:58.589861 27235 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a7bbada61b415c868dc235ca7c0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:58 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:55510
I20251025 20:08:58.589927 27377 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a7bbada61b415c868dc235ca7c0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:58 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:39540
I20251025 20:08:58.590435 27495 tablet_replica.cc:333] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: stopping tablet replica
I20251025 20:08:58.590435 27496 tablet_replica.cc:333] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: stopping tablet replica
I20251025 20:08:58.590696 27496 raft_consensus.cc:2243] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:58.591034 26952 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 20:08:58.591204 27496 raft_consensus.cc:2272] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 20:08:58.591461 26952 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:58.591753 27495 raft_consensus.cc:2243] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:58.592043 27495 raft_consensus.cc:2272] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:58.592614 27496 ts_tablet_manager.cc:1916] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:58.593369 27495 ts_tablet_manager.cc:1916] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:58.595224 27496 ts_tablet_manager.cc:1929] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:58.595368 27496 log.cc:1199] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/a2a7bbada61b415c868dc235ca7c0406
I20251025 20:08:58.595726 27496 ts_tablet_manager.cc:1950] T a2a7bbada61b415c868dc235ca7c0406 P 4c194dacee924f0b8e081f986bf09cbe: Deleting consensus metadata
I20251025 20:08:58.595968 27495 ts_tablet_manager.cc:1929] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:58.596053 27495 log.cc:1199] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/a2a7bbada61b415c868dc235ca7c0406
I20251025 20:08:58.596323 27495 ts_tablet_manager.cc:1950] T a2a7bbada61b415c868dc235ca7c0406 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting consensus metadata
I20251025 20:08:58.597151 27377 tablet_service.cc:1505] Processing CreateTablet for tablet 0a647a83316c48dd89334fb6578d6cca (DEFAULT_TABLE table=test-table [id=d177a37cc9014423880c2ccfe1068c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.597390 27494 tablet_replica.cc:333] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: stopping tablet replica
I20251025 20:08:58.597352 27093 tablet_service.cc:1505] Processing CreateTablet for tablet 0a647a83316c48dd89334fb6578d6cca (DEFAULT_TABLE table=test-table [id=d177a37cc9014423880c2ccfe1068c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.597448 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a647a83316c48dd89334fb6578d6cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.597608 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a647a83316c48dd89334fb6578d6cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.597648 27494 raft_consensus.cc:2243] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:58.597975 27494 raft_consensus.cc:2272] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:58.599172 27471 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Bootstrap starting.
I20251025 20:08:58.599462 26940 catalog_manager.cc:4985] TS 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361): tablet a2a7bbada61b415c868dc235ca7c0406 (table test-table [id=0fe7d54bd0264278be993c5b2841dc98]) successfully deleted
I20251025 20:08:58.599629 27472 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Bootstrap starting.
I20251025 20:08:58.599838 27235 tablet_service.cc:1505] Processing CreateTablet for tablet 0a647a83316c48dd89334fb6578d6cca (DEFAULT_TABLE table=test-table [id=d177a37cc9014423880c2ccfe1068c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:58.600080 27471 tablet_bootstrap.cc:654] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.600087 27235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a647a83316c48dd89334fb6578d6cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.600564 27472 tablet_bootstrap.cc:654] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.601045 27471 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: No bootstrap required, opened a new log
I20251025 20:08:58.601128 27471 ts_tablet_manager.cc:1403] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:58.601665 27471 raft_consensus.cc:359] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.601744 27472 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: No bootstrap required, opened a new log
I20251025 20:08:58.601760 27471 raft_consensus.cc:385] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.601792 27471 raft_consensus.cc:740] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.601815 27472 ts_tablet_manager.cc:1403] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:58.601915 27471 consensus_queue.cc:260] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.602145 27471 ts_tablet_manager.cc:1434] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:58.602248 27472 raft_consensus.cc:359] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.602356 27472 raft_consensus.cc:385] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.602392 27472 raft_consensus.cc:740] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.602511 27472 consensus_queue.cc:260] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.602734 27472 ts_tablet_manager.cc:1434] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 20:08:58.602907 26940 catalog_manager.cc:4985] TS ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159): tablet a2a7bbada61b415c868dc235ca7c0406 (table test-table [id=0fe7d54bd0264278be993c5b2841dc98]) successfully deleted
I20251025 20:08:58.603489 27473 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Bootstrap starting.
I20251025 20:08:58.604514 27473 tablet_bootstrap.cc:654] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.605016 27494 ts_tablet_manager.cc:1916] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:58.606043 27473 tablet_bootstrap.cc:492] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: No bootstrap required, opened a new log
I20251025 20:08:58.606118 27473 ts_tablet_manager.cc:1403] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:58.606591 27473 raft_consensus.cc:359] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.606685 27473 raft_consensus.cc:385] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.606714 27473 raft_consensus.cc:740] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.607322 27473 consensus_queue.cc:260] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.607692 27494 ts_tablet_manager.cc:1929] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:58.607824 27494 log.cc:1199] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/a2a7bbada61b415c868dc235ca7c0406
I20251025 20:08:58.608160 27494 ts_tablet_manager.cc:1950] T a2a7bbada61b415c868dc235ca7c0406 P a03af3799333416fb8640f01f8ee755f: Deleting consensus metadata
I20251025 20:08:58.608371 27473 ts_tablet_manager.cc:1434] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:58.609342 26940 catalog_manager.cc:4985] TS a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991): tablet a2a7bbada61b415c868dc235ca7c0406 (table test-table [id=0fe7d54bd0264278be993c5b2841dc98]) successfully deleted
I20251025 20:08:58.640230 27479 raft_consensus.cc:493] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:58.640367 27479 raft_consensus.cc:515] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.641362 27479 leader_election.cc:290] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:58.644394 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.641606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36159 (local address 127.18.82.66:36897)
1025 20:08:58.641733 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.641751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.641846 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.642233 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.642236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.642253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.642521 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.642532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.643440 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.643444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.644018 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.644025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.644134 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.644157 (+ 23us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.644209 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.644271 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20251025 20:08:58.644881 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.641685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:36897 (local address 127.18.82.65:36159)
1025 20:08:58.641817 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.641822 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.641835 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:08:58.641985 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.641989 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.642047 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.642116 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.642655 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.643322 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.644136 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.644605 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.644669 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.644718 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.644766 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:08:58.645153 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.641823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.18.82.66:45771)
1025 20:08:58.642115 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.642127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.642221 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.642489 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.642493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.642509 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:58.642823 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.642831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.643857 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.643861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.644770 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.644779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.644905 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.644924 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.644971 (+ 47us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.645024 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20251025 20:08:58.645226 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a647a83316c48dd89334fb6578d6cca" candidate_uuid: "a03af3799333416fb8640f01f8ee755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" is_pre_election: true
I20251025 20:08:58.645365 27113 raft_consensus.cc:2468] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03af3799333416fb8640f01f8ee755f in term 0.
I20251025 20:08:58.645565 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.642010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:45771 (local address 127.18.82.67:45361)
1025 20:08:58.642136 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.642140 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.642179 (+ 39us) server_negotiation.cc:408] Connection header received
1025 20:08:58.642334 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.642337 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.642401 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.642472 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:58.642967 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.643714 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.644876 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.645275 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.645311 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.645363 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.645420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 20:08:58.645751 27188 leader_election.cc:304] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:58.645946 27479 raft_consensus.cc:2804] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:58.645996 27479 raft_consensus.cc:493] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:58.646029 27479 raft_consensus.cc:3060] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.646081 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a647a83316c48dd89334fb6578d6cca" candidate_uuid: "a03af3799333416fb8640f01f8ee755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe" is_pre_election: true
I20251025 20:08:58.646297 27397 raft_consensus.cc:2468] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03af3799333416fb8640f01f8ee755f in term 0.
I20251025 20:08:58.647133 27479 raft_consensus.cc:515] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.647464 27479 leader_election.cc:290] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [CANDIDATE]: Term 1 election: Requested vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:58.647773 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a647a83316c48dd89334fb6578d6cca" candidate_uuid: "a03af3799333416fb8640f01f8ee755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
I20251025 20:08:58.647886 27113 raft_consensus.cc:3060] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.647881 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a647a83316c48dd89334fb6578d6cca" candidate_uuid: "a03af3799333416fb8640f01f8ee755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe"
I20251025 20:08:58.647974 27397 raft_consensus.cc:3060] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.648972 27113 raft_consensus.cc:2468] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03af3799333416fb8640f01f8ee755f in term 1.
I20251025 20:08:58.648972 27397 raft_consensus.cc:2468] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03af3799333416fb8640f01f8ee755f in term 1.
I20251025 20:08:58.649250 27188 leader_election.cc:304] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:58.649430 27479 raft_consensus.cc:2804] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:58.649667 27479 raft_consensus.cc:697] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 1 LEADER]: Becoming Leader. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Running, Role: LEADER
I20251025 20:08:58.649911 27479 consensus_queue.cc:237] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.652539 26952 catalog_manager.cc:5649] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f reported cstate change: term changed from 0 to 1, leader changed from <none> to a03af3799333416fb8640f01f8ee755f (127.18.82.66). New cstate: current_term: 1 leader_uuid: "a03af3799333416fb8640f01f8ee755f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } health_report { overall_health: UNKNOWN } } }
W20251025 20:08:58.697448 27444 tablet.cc:2378] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 25 20:08:58 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422937, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.126@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:58.707285 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.699619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:40179 (local address 127.0.0.1:50558)
1025 20:08:58.699949 (+ 330us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:58.699960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:58.700080 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:58.700284 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:58.700286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:58.700705 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:08:58.700892 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.700898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.701750 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.701754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:58.702149 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:58.702158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.702251 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.702805 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:08:58.702817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:08:58.704121 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:08:58.705943 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.705949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.705953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.706185 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.706462 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:08:58.706465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:08:58.706466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:08:58.706515 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:08:58.706878 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:08:58.706883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:08:58.706976 (+ 93us) client_negotiation.cc:769] Sending connection context
1025 20:08:58.707062 (+ 86us) client_negotiation.cc:240] Negotiation successful
1025 20:08:58.707158 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20251025 20:08:58.707429 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:58.699662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.18.82.126:40179)
1025 20:08:58.699796 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:58.699801 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:58.700034 (+ 233us) server_negotiation.cc:408] Connection header received
1025 20:08:58.700157 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:58.700161 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:58.700195 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:58.700259 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:58.701003 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.701474 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:58.702274 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:58.702462 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:58.704246 (+ 1784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:58.704263 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:58.704267 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:58.704294 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:58.705812 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.706287 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.706291 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.706293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.706362 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:58.706596 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:58.706599 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:58.706601 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:58.706777 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:58.706879 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:58.707152 (+ 273us) server_negotiation.cc:300] Negotiation successful
1025 20:08:58.707264 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:08:58.711254 26952 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:50558:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 20:08:58.711997 26952 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:58.717219 27093 tablet_service.cc:1505] Processing CreateTablet for tablet 5923d947446e4e69b0c034aacf6ee02a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=234f4f98861f4e3e87f88c6c609b1d0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:58.717356 27377 tablet_service.cc:1505] Processing CreateTablet for tablet 5923d947446e4e69b0c034aacf6ee02a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=234f4f98861f4e3e87f88c6c609b1d0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:58.717703 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923d947446e4e69b0c034aacf6ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.717728 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923d947446e4e69b0c034aacf6ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.719763 27471 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: Bootstrap starting.
I20251025 20:08:58.720567 27472 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: Bootstrap starting.
I20251025 20:08:58.720841 27471 tablet_bootstrap.cc:654] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.720954 27235 tablet_service.cc:1505] Processing CreateTablet for tablet 5923d947446e4e69b0c034aacf6ee02a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=234f4f98861f4e3e87f88c6c609b1d0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 20:08:58.721310 27235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923d947446e4e69b0c034aacf6ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:58.721647 27472 tablet_bootstrap.cc:654] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.722237 27471 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: No bootstrap required, opened a new log
I20251025 20:08:58.722325 27471 ts_tablet_manager.cc:1403] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:58.722738 27471 raft_consensus.cc:359] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.722826 27471 raft_consensus.cc:385] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.722844 27471 raft_consensus.cc:740] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.722945 27471 consensus_queue.cc:260] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.723165 27471 ts_tablet_manager.cc:1434] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 20:08:58.723142 27477 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:58.723249 27477 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: This TxnStatusTablet replica's current role is: FOLLOWER
W20251025 20:08:58.723726 27302 tablet.cc:2378] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:58.724277 27473 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: Bootstrap starting.
I20251025 20:08:58.725235 27473 tablet_bootstrap.cc:654] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:58.725898 27472 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: No bootstrap required, opened a new log
I20251025 20:08:58.725981 27472 ts_tablet_manager.cc:1403] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251025 20:08:58.726531 27473 tablet_bootstrap.cc:492] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: No bootstrap required, opened a new log
I20251025 20:08:58.726621 27473 ts_tablet_manager.cc:1403] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 20:08:58.726563 27472 raft_consensus.cc:359] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.726658 27472 raft_consensus.cc:385] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.726688 27472 raft_consensus.cc:740] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.726811 27472 consensus_queue.cc:260] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.727090 27473 raft_consensus.cc:359] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.727193 27473 raft_consensus.cc:385] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:58.727228 27473 raft_consensus.cc:740] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Initialized, Role: FOLLOWER
I20251025 20:08:58.727345 27473 consensus_queue.cc:260] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.727327 27478 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:58.727455 27478 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:58.727622 27473 ts_tablet_manager.cc:1434] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:58.728632 27472 ts_tablet_manager.cc:1434] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251025 20:08:58.730255 27479 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:58.730394 27479 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: This TxnStatusTablet replica's current role is: FOLLOWER
W20251025 20:08:58.759708 27160 tablet.cc:2378] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:08:58.979593 27477 raft_consensus.cc:493] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:58.979729 27477 raft_consensus.cc:515] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.980160 27477 leader_election.cc:290] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991)
I20251025 20:08:58.980636 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5923d947446e4e69b0c034aacf6ee02a" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" is_pre_election: true
I20251025 20:08:58.980669 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5923d947446e4e69b0c034aacf6ee02a" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f" is_pre_election: true
I20251025 20:08:58.980793 27113 raft_consensus.cc:2468] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 0.
I20251025 20:08:58.980793 27255 raft_consensus.cc:2468] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 0.
I20251025 20:08:58.981096 27330 leader_election.cc:304] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c194dacee924f0b8e081f986bf09cbe, a03af3799333416fb8640f01f8ee755f; no voters:
I20251025 20:08:58.981256 27477 raft_consensus.cc:2804] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:58.981313 27477 raft_consensus.cc:493] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:58.981379 27477 raft_consensus.cc:3060] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.982270 27477 raft_consensus.cc:515] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.982579 27477 leader_election.cc:290] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 election: Requested vote from peers ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159), a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991)
I20251025 20:08:58.982968 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5923d947446e4e69b0c034aacf6ee02a" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f"
I20251025 20:08:58.983016 27113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5923d947446e4e69b0c034aacf6ee02a" candidate_uuid: "4c194dacee924f0b8e081f986bf09cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec"
I20251025 20:08:58.983105 27255 raft_consensus.cc:3060] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.983121 27113 raft_consensus.cc:3060] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:58.984114 27113 raft_consensus.cc:2468] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 1.
I20251025 20:08:58.984114 27255 raft_consensus.cc:2468] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c194dacee924f0b8e081f986bf09cbe in term 1.
I20251025 20:08:58.984404 27330 leader_election.cc:304] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c194dacee924f0b8e081f986bf09cbe, a03af3799333416fb8640f01f8ee755f; no voters:
I20251025 20:08:58.984548 27477 raft_consensus.cc:2804] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:58.984635 27477 raft_consensus.cc:697] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [term 1 LEADER]: Becoming Leader. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Running, Role: LEADER
I20251025 20:08:58.984812 27477 consensus_queue.cc:237] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:58.985381 27477 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: TxnStatusTablet state changed. Reason: New leader 4c194dacee924f0b8e081f986bf09cbe. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:58.985459 27477 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:58.985684 27513 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 20:08:58.985816 27513 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 20:08:58.986548 26953 catalog_manager.cc:5649] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67). New cstate: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } health_report { overall_health: HEALTHY } } }
I20251025 20:08:59.025703 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.022945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.18.82.67:45361)
1025 20:08:59.023256 (+ 311us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.023259 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.023269 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:08:59.023309 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.023311 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.023367 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.023427 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.023821 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.024329 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.025009 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.025201 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.025243 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.025313 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.025387 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.025503 (+ 116us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.025574 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20251025 20:08:59.025703 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.022852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.0.0.1:39558)
1025 20:08:59.023010 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.023024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.023115 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.023463 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.023465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.023473 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:59.023708 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.023713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.024444 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.024447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.024893 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.024901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.024999 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.025047 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.025413 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.025427 (+ 14us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.025485 (+ 58us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.025580 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20251025 20:08:59.028338 27255 raft_consensus.cc:1275] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Refusing update from remote peer 4c194dacee924f0b8e081f986bf09cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.028340 27113 raft_consensus.cc:1275] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Refusing update from remote peer 4c194dacee924f0b8e081f986bf09cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.028771 27489 consensus_queue.cc:1048] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.028930 27477 consensus_queue.cc:1048] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.030256 27478 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: TxnStatusTablet state changed. Reason: New leader 4c194dacee924f0b8e081f986bf09cbe. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.030359 27478 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:59.031932 27489 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.032042 27489 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:59.032224 27479 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: TxnStatusTablet state changed. Reason: New leader 4c194dacee924f0b8e081f986bf09cbe. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.032323 27479 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:59.032756 27484 mvcc.cc:204] Tried to move back new op lower bound from 7214788358255869952 to 7214788358082887680. Current Snapshot: MvccSnapshot[applied={T|T < 7214788358255869952 or (T in {7214788358255869952})}]
I20251025 20:08:59.033547 27479 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.033656 27479 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P a03af3799333416fb8640f01f8ee755f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:59.035152 27477 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.035238 27477 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P 4c194dacee924f0b8e081f986bf09cbe: This TxnStatusTablet replica's current role is: LEADER
I20251025 20:08:59.036110 27478 tablet_replica.cc:442] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c194dacee924f0b8e081f986bf09cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } } }
I20251025 20:08:59.036203 27478 tablet_replica.cc:445] T 5923d947446e4e69b0c034aacf6ee02a P ab5fac3d4cf245d89b3b0f3928d8f6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 20:08:59.041795 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.038432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.18.82.66:37991)
1025 20:08:59.038802 (+ 370us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.038806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.038815 (+ 9us) server_negotiation.cc:408] Connection header received
1025 20:08:59.038890 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.038893 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.038956 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.039053 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.039369 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.039813 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.040423 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.040593 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.040645 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.041183 (+ 538us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.041291 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.041619 (+ 328us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.041675 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20251025 20:08:59.041755 27479 consensus_queue.cc:1048] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.045434 27479 consensus_queue.cc:1048] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.052314 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.049378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:50659 (local address 127.18.82.67:45361)
1025 20:08:59.049487 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.049494 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.049531 (+ 37us) server_negotiation.cc:408] Connection header received
1025 20:08:59.049573 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.049576 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.049617 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.049673 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.050187 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.050921 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.051821 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.051951 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.051988 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.052047 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.052106 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.052193 (+ 87us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.052227 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251025 20:08:59.052383 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.049066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.18.82.66:50659)
1025 20:08:59.049360 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.049375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.049465 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.049699 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.049702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.049709 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:59.050061 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.050071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.051054 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.051058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.051721 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.051731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.051839 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.051878 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.052120 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.052125 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.052188 (+ 63us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.052247 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20251025 20:08:59.075147 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.072082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:50093 (local address 127.18.82.66:37991)
1025 20:08:59.072202 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.072206 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.072342 (+ 136us) server_negotiation.cc:408] Connection header received
1025 20:08:59.072450 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.072453 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.072504 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.072593 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.072967 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.073753 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.074607 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.074751 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.074790 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.074848 (+ 58us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.074907 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.075010 (+ 103us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.075048 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:08:59.075294 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.072008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37991 (local address 127.18.82.67:50093)
1025 20:08:59.072269 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.072282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.072369 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.072589 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.072595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.072602 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:59.072862 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.072870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.073886 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.073889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.074495 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.074505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.074632 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.074679 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.074936 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.074942 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.075004 (+ 62us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.075073 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20251025 20:08:59.100661 26953 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.100845 26953 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.103053 26953 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Sending DeleteTablet for 3 replicas of tablet 0a647a83316c48dd89334fb6578d6cca
I20251025 20:08:59.103803 27235 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a647a83316c48dd89334fb6578d6cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:55510
I20251025 20:08:59.103991 27494 tablet_replica.cc:333] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: stopping tablet replica
I20251025 20:08:59.104110 27494 raft_consensus.cc:2243] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:59.104276 27377 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a647a83316c48dd89334fb6578d6cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:39540
I20251025 20:08:59.104411 27494 raft_consensus.cc:2272] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.104779 27093 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a647a83316c48dd89334fb6578d6cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37928
I20251025 20:08:59.105142 27530 tablet_replica.cc:333] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: stopping tablet replica
I20251025 20:08:59.105289 27530 raft_consensus.cc:2243] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.105445 27530 raft_consensus.cc:2272] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.106191 27530 ts_tablet_manager.cc:1916] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.106232 27494 ts_tablet_manager.cc:1916] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.108956 27494 ts_tablet_manager.cc:1929] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.108966 27530 ts_tablet_manager.cc:1929] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.109021 27494 log.cc:1199] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/0a647a83316c48dd89334fb6578d6cca
I20251025 20:08:59.109033 27530 log.cc:1199] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/0a647a83316c48dd89334fb6578d6cca
I20251025 20:08:59.109278 27494 ts_tablet_manager.cc:1950] T 0a647a83316c48dd89334fb6578d6cca P a03af3799333416fb8640f01f8ee755f: Deleting consensus metadata
I20251025 20:08:59.109300 27530 ts_tablet_manager.cc:1950] T 0a647a83316c48dd89334fb6578d6cca P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting consensus metadata
I20251025 20:08:59.109665 27529 tablet_replica.cc:333] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: stopping tablet replica
I20251025 20:08:59.109815 27529 raft_consensus.cc:2243] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.109998 27529 raft_consensus.cc:2272] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.110044 26940 catalog_manager.cc:4985] TS ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159): tablet 0a647a83316c48dd89334fb6578d6cca (table test-table [id=d177a37cc9014423880c2ccfe1068c59]) successfully deleted
I20251025 20:08:59.112349 26940 catalog_manager.cc:4985] TS a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991): tablet 0a647a83316c48dd89334fb6578d6cca (table test-table [id=d177a37cc9014423880c2ccfe1068c59]) successfully deleted
I20251025 20:08:59.118013 27529 ts_tablet_manager.cc:1916] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.120716 27529 ts_tablet_manager.cc:1929] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.120846 27529 log.cc:1199] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/0a647a83316c48dd89334fb6578d6cca
I20251025 20:08:59.121141 27529 ts_tablet_manager.cc:1950] T 0a647a83316c48dd89334fb6578d6cca P 4c194dacee924f0b8e081f986bf09cbe: Deleting consensus metadata
I20251025 20:08:59.121986 26940 catalog_manager.cc:4985] TS 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361): tablet 0a647a83316c48dd89334fb6578d6cca (table test-table [id=d177a37cc9014423880c2ccfe1068c59]) successfully deleted
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[26906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
W20251025 20:08:59.139679 27003 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:08:59.132457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.18.82.126:40179)
1025 20:08:59.132615 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.132620 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.132879 (+ 259us) server_negotiation.cc:408] Connection header received
1025 20:08:59.133018 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.133021 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.133097 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.133182 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:59.134360 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.135169 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.136052 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.136231 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.139545 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50560: BlockingRecv error: recv got EOF from 127.0.0.1:50560 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[26906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM
I20251025 20:08:59.150833 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.141643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.18.82.126:40179)
1025 20:08:59.141754 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.141758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.142172 (+ 414us) server_negotiation.cc:408] Connection header received
1025 20:08:59.142305 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.142308 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.142360 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.142436 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:08:59.143257 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.144027 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.144869 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.145082 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.147441 (+ 2359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:08:59.147460 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:08:59.147464 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:08:59.147491 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:08:59.149149 (+ 1658us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:59.149639 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:59.149643 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:59.149645 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:59.149698 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:08:59.150030 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:08:59.150034 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:08:59.150036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:08:59.150205 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:08:59.150342 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.150571 (+ 229us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.150681 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:08:59.153277 26953 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:59.153817 26953 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:59.159221 27093 tablet_service.cc:1505] Processing CreateTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 (DEFAULT_TABLE table=test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.159518 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8681d0e9604d468be5d7fcf82bbe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.159848 27235 tablet_service.cc:1505] Processing CreateTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 (DEFAULT_TABLE table=test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.160121 27235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8681d0e9604d468be5d7fcf82bbe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.161777 27472 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Bootstrap starting.
I20251025 20:08:59.161825 27377 tablet_service.cc:1505] Processing CreateTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 (DEFAULT_TABLE table=test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.162099 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8681d0e9604d468be5d7fcf82bbe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.162674 27472 tablet_bootstrap.cc:654] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.164212 27472 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: No bootstrap required, opened a new log
I20251025 20:08:59.164296 27472 ts_tablet_manager.cc:1403] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251025 20:08:59.164762 27472 raft_consensus.cc:359] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.164870 27472 raft_consensus.cc:385] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.164897 27472 raft_consensus.cc:740] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.164999 27472 consensus_queue.cc:260] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.165220 27472 ts_tablet_manager.cc:1434] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 20:08:59.165427 27471 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Bootstrap starting.
I20251025 20:08:59.165433 27473 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Bootstrap starting.
I20251025 20:08:59.166369 27471 tablet_bootstrap.cc:654] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.166433 27473 tablet_bootstrap.cc:654] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.168010 27471 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: No bootstrap required, opened a new log
I20251025 20:08:59.168093 27471 ts_tablet_manager.cc:1403] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 20:08:59.168602 27471 raft_consensus.cc:359] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.168704 27471 raft_consensus.cc:385] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.168730 27471 raft_consensus.cc:740] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.168839 27471 consensus_queue.cc:260] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.169068 27471 ts_tablet_manager.cc:1434] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:59.174340 27473 tablet_bootstrap.cc:492] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: No bootstrap required, opened a new log
I20251025 20:08:59.174432 27473 ts_tablet_manager.cc:1403] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251025 20:08:59.174958 27473 raft_consensus.cc:359] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.175060 27473 raft_consensus.cc:385] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.175096 27473 raft_consensus.cc:740] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.175232 27473 consensus_queue.cc:260] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.175482 27473 ts_tablet_manager.cc:1434] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:59.308185 27521 raft_consensus.cc:493] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:59.308334 27521 raft_consensus.cc:515] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.309412 27521 leader_election.cc:290] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:59.313026 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.309660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:37991 (local address 127.18.82.65:53645)
1025 20:08:59.309915 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.309933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.310052 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.310345 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.310349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.310365 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:59.310642 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.310650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.311636 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.311642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.312584 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.312594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.312727 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.312738 (+ 11us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.312790 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.312851 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20251025 20:08:59.313234 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.310038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.18.82.65:52713)
1025 20:08:59.310250 (+ 212us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.310260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.310345 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.310563 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.310566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.310580 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:08:59.310793 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.310798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.311657 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.311659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.312584 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.312594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.312713 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.312738 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.312790 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.312851 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20251025 20:08:59.313417 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.310135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:52713 (local address 127.18.82.67:45361)
1025 20:08:59.310259 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.310265 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.310325 (+ 60us) server_negotiation.cc:408] Connection header received
1025 20:08:59.310428 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.310432 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.310474 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.310533 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:59.310877 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.311571 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.312712 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.313164 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.313200 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.313250 (+ 50us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.313300 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:08:59.313417 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.309742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:53645 (local address 127.18.82.66:37991)
1025 20:08:59.309871 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.309875 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.310001 (+ 126us) server_negotiation.cc:408] Connection header received
1025 20:08:59.310161 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.310165 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.310225 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.310323 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:08:59.310758 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.311490 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.312712 (+ 1222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.313164 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.313200 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.313251 (+ 51us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.313300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 20:08:59.313875 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd8681d0e9604d468be5d7fcf82bbe09" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f" is_pre_election: true
I20251025 20:08:59.313890 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd8681d0e9604d468be5d7fcf82bbe09" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe" is_pre_election: true
I20251025 20:08:59.313999 27255 raft_consensus.cc:2468] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 0.
I20251025 20:08:59.314014 27397 raft_consensus.cc:2468] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 0.
I20251025 20:08:59.314334 27046 leader_election.cc:304] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:59.314549 27521 raft_consensus.cc:2804] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:59.314605 27521 raft_consensus.cc:493] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:59.314633 27521 raft_consensus.cc:3060] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.315522 27521 raft_consensus.cc:515] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.315821 27521 leader_election.cc:290] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 election: Requested vote from peers a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:59.316098 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd8681d0e9604d468be5d7fcf82bbe09" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f"
I20251025 20:08:59.316190 27255 raft_consensus.cc:3060] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.316169 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd8681d0e9604d468be5d7fcf82bbe09" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe"
I20251025 20:08:59.316329 27397 raft_consensus.cc:3060] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.316962 27255 raft_consensus.cc:2468] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 1.
I20251025 20:08:59.317091 27397 raft_consensus.cc:2468] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 1.
I20251025 20:08:59.317219 27046 leader_election.cc:304] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:59.317458 27521 raft_consensus.cc:2804] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:59.317579 27521 raft_consensus.cc:697] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 LEADER]: Becoming Leader. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Running, Role: LEADER
I20251025 20:08:59.317804 27521 consensus_queue.cc:237] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.319957 26953 catalog_manager.cc:5649] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65). New cstate: current_term: 1 leader_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:59.361757 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.358565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.18.82.65:36159)
1025 20:08:59.358851 (+ 286us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.358855 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.358865 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:08:59.358945 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.358947 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.358998 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.359096 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.359480 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.359986 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.360652 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.360824 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.360881 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.361343 (+ 462us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.361454 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.361535 (+ 81us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.361591 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20251025 20:08:59.364225 27397 raft_consensus.cc:1275] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Refusing update from remote peer ab5fac3d4cf245d89b3b0f3928d8f6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.364305 27255 raft_consensus.cc:1275] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Refusing update from remote peer ab5fac3d4cf245d89b3b0f3928d8f6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.364670 27521 consensus_queue.cc:1048] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.364826 27478 consensus_queue.cc:1048] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.375327 26953 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.375450 26953 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.377305 26953 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Sending DeleteTablet for 3 replicas of tablet dd8681d0e9604d468be5d7fcf82bbe09
I20251025 20:08:59.377791 27093 tablet_service.cc:1552] Processing DeleteTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37928
I20251025 20:08:59.377943 27530 tablet_replica.cc:333] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: stopping tablet replica
I20251025 20:08:59.377914 27235 tablet_service.cc:1552] Processing DeleteTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:55510
I20251025 20:08:59.378010 27377 tablet_service.cc:1552] Processing DeleteTablet for tablet dd8681d0e9604d468be5d7fcf82bbe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:39540
I20251025 20:08:59.378046 27530 raft_consensus.cc:2243] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:59.378193 27494 tablet_replica.cc:333] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: stopping tablet replica
I20251025 20:08:59.378274 27530 raft_consensus.cc:2272] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.378309 27494 raft_consensus.cc:2243] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.378435 27494 raft_consensus.cc:2272] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.378594 27529 tablet_replica.cc:333] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: stopping tablet replica
I20251025 20:08:59.378701 27529 raft_consensus.cc:2243] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.378805 27529 raft_consensus.cc:2272] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.378918 27530 ts_tablet_manager.cc:1916] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.379598 27529 ts_tablet_manager.cc:1916] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.379701 26953 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:08:59.380194 26953 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:08:59.381203 27530 ts_tablet_manager.cc:1929] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:59.381268 27530 log.cc:1199] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/dd8681d0e9604d468be5d7fcf82bbe09
I20251025 20:08:59.381502 27530 ts_tablet_manager.cc:1950] T dd8681d0e9604d468be5d7fcf82bbe09 P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting consensus metadata
I20251025 20:08:59.381959 27529 ts_tablet_manager.cc:1929] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:59.382033 27529 log.cc:1199] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/dd8681d0e9604d468be5d7fcf82bbe09
I20251025 20:08:59.382278 27529 ts_tablet_manager.cc:1950] T dd8681d0e9604d468be5d7fcf82bbe09 P 4c194dacee924f0b8e081f986bf09cbe: Deleting consensus metadata
I20251025 20:08:59.383066 26940 catalog_manager.cc:4985] TS ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159): tablet dd8681d0e9604d468be5d7fcf82bbe09 (table test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]) successfully deleted
I20251025 20:08:59.383107 27494 ts_tablet_manager.cc:1916] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.383662 26940 catalog_manager.cc:4985] TS 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361): tablet dd8681d0e9604d468be5d7fcf82bbe09 (table test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]) successfully deleted
I20251025 20:08:59.384861 27494 ts_tablet_manager.cc:1929] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:08:59.384924 27494 log.cc:1199] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/dd8681d0e9604d468be5d7fcf82bbe09
I20251025 20:08:59.385129 27494 ts_tablet_manager.cc:1950] T dd8681d0e9604d468be5d7fcf82bbe09 P a03af3799333416fb8640f01f8ee755f: Deleting consensus metadata
I20251025 20:08:59.385984 26940 catalog_manager.cc:4985] TS a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991): tablet dd8681d0e9604d468be5d7fcf82bbe09 (table test-table [id=be5af0d3a11140e1adbdfb24e5b85aeb]) successfully deleted
I20251025 20:08:59.386847 27093 tablet_service.cc:1505] Processing CreateTablet for tablet 929dadef83304b10afc631313b5b8b3f (DEFAULT_TABLE table=test-table [id=8b8764d8305944688524367c05c0b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.386930 27235 tablet_service.cc:1505] Processing CreateTablet for tablet 929dadef83304b10afc631313b5b8b3f (DEFAULT_TABLE table=test-table [id=8b8764d8305944688524367c05c0b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.387097 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929dadef83304b10afc631313b5b8b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.387127 27235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929dadef83304b10afc631313b5b8b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.387177 27377 tablet_service.cc:1505] Processing CreateTablet for tablet 929dadef83304b10afc631313b5b8b3f (DEFAULT_TABLE table=test-table [id=8b8764d8305944688524367c05c0b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:08:59.387424 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929dadef83304b10afc631313b5b8b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:08:59.389263 27472 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Bootstrap starting.
I20251025 20:08:59.389427 27471 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Bootstrap starting.
I20251025 20:08:59.389528 27473 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Bootstrap starting.
I20251025 20:08:59.390218 27471 tablet_bootstrap.cc:654] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.390218 27472 tablet_bootstrap.cc:654] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.390352 27473 tablet_bootstrap.cc:654] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Neither blocks nor log segments found. Creating new log.
I20251025 20:08:59.391317 27471 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: No bootstrap required, opened a new log
I20251025 20:08:59.391429 27471 ts_tablet_manager.cc:1403] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:59.391777 27473 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: No bootstrap required, opened a new log
I20251025 20:08:59.391866 27473 ts_tablet_manager.cc:1403] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 20:08:59.391914 27472 tablet_bootstrap.cc:492] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: No bootstrap required, opened a new log
I20251025 20:08:59.391902 27471 raft_consensus.cc:359] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.391981 27472 ts_tablet_manager.cc:1403] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251025 20:08:59.392005 27471 raft_consensus.cc:385] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.392038 27471 raft_consensus.cc:740] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c194dacee924f0b8e081f986bf09cbe, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.392164 27471 consensus_queue.cc:260] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.392237 27473 raft_consensus.cc:359] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.392328 27473 raft_consensus.cc:385] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.392370 27473 raft_consensus.cc:740] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03af3799333416fb8640f01f8ee755f, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.392426 27471 ts_tablet_manager.cc:1434] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:59.392400 27472 raft_consensus.cc:359] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.392505 27472 raft_consensus.cc:385] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:08:59.392479 27473 consensus_queue.cc:260] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.392541 27472 raft_consensus.cc:740] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Initialized, Role: FOLLOWER
I20251025 20:08:59.392663 27472 consensus_queue.cc:260] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.392704 27473 ts_tablet_manager.cc:1434] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 20:08:59.392884 27472 ts_tablet_manager.cc:1434] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 20:08:59.539527 27553 raft_consensus.cc:493] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:08:59.539708 27553 raft_consensus.cc:515] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.540237 27553 leader_election.cc:290] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:59.540740 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "929dadef83304b10afc631313b5b8b3f" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f" is_pre_election: true
I20251025 20:08:59.540763 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "929dadef83304b10afc631313b5b8b3f" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe" is_pre_election: true
I20251025 20:08:59.540884 27255 raft_consensus.cc:2468] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 0.
I20251025 20:08:59.540894 27397 raft_consensus.cc:2468] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 0.
I20251025 20:08:59.541215 27046 leader_election.cc:304] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:59.541417 27553 raft_consensus.cc:2804] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:08:59.541481 27553 raft_consensus.cc:493] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:08:59.541540 27553 raft_consensus.cc:3060] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.542569 27553 raft_consensus.cc:515] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.542899 27553 leader_election.cc:290] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 election: Requested vote from peers a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991), 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361)
I20251025 20:08:59.543233 27255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "929dadef83304b10afc631313b5b8b3f" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03af3799333416fb8640f01f8ee755f"
I20251025 20:08:59.543246 27397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "929dadef83304b10afc631313b5b8b3f" candidate_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c194dacee924f0b8e081f986bf09cbe"
I20251025 20:08:59.543356 27397 raft_consensus.cc:3060] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.543352 27255 raft_consensus.cc:3060] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:08:59.544179 27255 raft_consensus.cc:2468] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 1.
I20251025 20:08:59.544437 27397 raft_consensus.cc:2468] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5fac3d4cf245d89b3b0f3928d8f6ec in term 1.
I20251025 20:08:59.544479 27046 leader_election.cc:304] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03af3799333416fb8640f01f8ee755f, ab5fac3d4cf245d89b3b0f3928d8f6ec; no voters:
I20251025 20:08:59.544672 27553 raft_consensus.cc:2804] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:08:59.544778 27553 raft_consensus.cc:697] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 LEADER]: Becoming Leader. State: Replica: ab5fac3d4cf245d89b3b0f3928d8f6ec, State: Running, Role: LEADER
I20251025 20:08:59.544929 27553 consensus_queue.cc:237] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } }
I20251025 20:08:59.546492 26953 catalog_manager.cc:5649] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65). New cstate: current_term: 1 leader_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5fac3d4cf245d89b3b0f3928d8f6ec" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 36159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 } health_report { overall_health: UNKNOWN } } }
I20251025 20:08:59.598934 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.596740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:50523 (local address 127.18.82.67:45361)
1025 20:08:59.596844 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.596848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.596978 (+ 130us) server_negotiation.cc:408] Connection header received
1025 20:08:59.597075 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.597078 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.597117 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.597173 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.597446 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.597870 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.598398 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.598511 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.598544 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.598619 (+ 75us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.598680 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.598776 (+ 96us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.598833 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 20:08:59.598965 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.596666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:45361 (local address 127.18.82.65:50523)
1025 20:08:59.596920 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.596931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.597013 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.597182 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.597183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.597188 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:59.597374 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.597378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.597938 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.597940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.598306 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.598312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.598415 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.598462 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.598702 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.598707 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.598755 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.598831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20251025 20:08:59.603717 27255 raft_consensus.cc:1275] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Refusing update from remote peer ab5fac3d4cf245d89b3b0f3928d8f6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.603717 27397 raft_consensus.cc:1275] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Refusing update from remote peer ab5fac3d4cf245d89b3b0f3928d8f6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:08:59.604084 27553 consensus_queue.cc:1048] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03af3799333416fb8640f01f8ee755f" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.604230 27521 consensus_queue.cc:1048] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c194dacee924f0b8e081f986bf09cbe" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 45361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:08:59.606487 27485 mvcc.cc:204] Tried to move back new op lower bound from 7214788360613576704 to 7214788360377430016. Current Snapshot: MvccSnapshot[applied={T|T < 7214788360613576704}]
I20251025 20:08:59.619001 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.616531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36159 (local address 127.18.82.67:33319)
1025 20:08:59.616752 (+ 221us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.616768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:08:59.616873 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:08:59.617137 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:08:59.617140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:08:59.617149 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 20:08:59.617366 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.617372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.618013 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.618015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:08:59.618360 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:08:59.618365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.618443 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.618464 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.618750 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.618756 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 20:08:59.618826 (+ 70us) client_negotiation.cc:240] Negotiation successful
1025 20:08:59.618905 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20251025 20:08:59.618994 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.616606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:33319 (local address 127.18.82.65:36159)
1025 20:08:59.616764 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1025 20:08:59.616769 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:08:59.616848 (+ 79us) server_negotiation.cc:408] Connection header received
1025 20:08:59.616976 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:08:59.616979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:08:59.617037 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:08:59.617119 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:08:59.617458 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.617942 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:08:59.618436 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:08:59.618566 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:08:59.618601 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:08:59.618664 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:08:59.618737 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1025 20:08:59.618839 (+ 102us) server_negotiation.cc:300] Negotiation successful
1025 20:08:59.618880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20251025 20:08:59.637624 26953 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.637754 26953 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50562:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:08:59.639374 26953 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3e9112ed74d6496b8c872a3f16732d90: Sending DeleteTablet for 3 replicas of tablet 929dadef83304b10afc631313b5b8b3f
I20251025 20:08:59.639835 27093 tablet_service.cc:1552] Processing DeleteTablet for tablet 929dadef83304b10afc631313b5b8b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37928
I20251025 20:08:59.639947 27377 tablet_service.cc:1552] Processing DeleteTablet for tablet 929dadef83304b10afc631313b5b8b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:39540
I20251025 20:08:59.639966 27235 tablet_service.cc:1552] Processing DeleteTablet for tablet 929dadef83304b10afc631313b5b8b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:08:59 UTC) from {username='oryx', principal='oryx/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:55510
I20251025 20:08:59.640004 27530 tablet_replica.cc:333] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: stopping tablet replica
I20251025 20:08:59.640115 27529 tablet_replica.cc:333] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: stopping tablet replica
I20251025 20:08:59.640120 27494 tablet_replica.cc:333] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: stopping tablet replica
I20251025 20:08:59.640107 27530 raft_consensus.cc:2243] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:08:59.640215 27494 raft_consensus.cc:2243] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.640229 27529 raft_consensus.cc:2243] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:08:59.640280 27530 raft_consensus.cc:2272] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.640321 27494 raft_consensus.cc:2272] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.640347 27529 raft_consensus.cc:2272] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:08:59.640988 27494 ts_tablet_manager.cc:1916] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.641010 27530 ts_tablet_manager.cc:1916] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.641010 27529 ts_tablet_manager.cc:1916] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:08:59.643007 27494 ts_tablet_manager.cc:1929] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.643075 27494 log.cc:1199] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/929dadef83304b10afc631313b5b8b3f
I20251025 20:08:59.643221 27530 ts_tablet_manager.cc:1929] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.643221 27529 ts_tablet_manager.cc:1929] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 20:08:59.643292 27530 log.cc:1199] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-0/wal/wals/929dadef83304b10afc631313b5b8b3f
I20251025 20:08:59.643293 27529 log.cc:1199] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/929dadef83304b10afc631313b5b8b3f
I20251025 20:08:59.643345 27494 ts_tablet_manager.cc:1950] T 929dadef83304b10afc631313b5b8b3f P a03af3799333416fb8640f01f8ee755f: Deleting consensus metadata
I20251025 20:08:59.643543 27530 ts_tablet_manager.cc:1950] T 929dadef83304b10afc631313b5b8b3f P ab5fac3d4cf245d89b3b0f3928d8f6ec: Deleting consensus metadata
I20251025 20:08:59.643543 27529 ts_tablet_manager.cc:1950] T 929dadef83304b10afc631313b5b8b3f P 4c194dacee924f0b8e081f986bf09cbe: Deleting consensus metadata
I20251025 20:08:59.643605 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27028
I20251025 20:08:59.645375 26940 catalog_manager.cc:4985] TS a03af3799333416fb8640f01f8ee755f (127.18.82.66:37991): tablet 929dadef83304b10afc631313b5b8b3f (table test-table [id=8b8764d8305944688524367c05c0b2a2]) successfully deleted
I20251025 20:08:59.645956 26940 catalog_manager.cc:4985] TS 4c194dacee924f0b8e081f986bf09cbe (127.18.82.67:45361): tablet 929dadef83304b10afc631313b5b8b3f (table test-table [id=8b8764d8305944688524367c05c0b2a2]) successfully deleted
W20251025 20:08:59.651938 26940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:36159 (error 108)
I20251025 20:08:59.651999 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27170
I20251025 20:08:59.653620 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:08:59.652932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:36159 (local address 127.0.0.1:37946)
1025 20:08:59.653426 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1025 20:08:59.653491 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:36159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":55,"threads_started":1}
W20251025 20:08:59.653753 26940 catalog_manager.cc:4712] TS ab5fac3d4cf245d89b3b0f3928d8f6ec (127.18.82.65:36159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 929dadef83304b10afc631313b5b8b3f: Network error: Client connection negotiation failed: client connection to 127.18.82.65:36159: connect: Connection refused (error 111)
I20251025 20:08:59.659027 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27312
I20251025 20:08:59.666148 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 26922
2025-10-25T20:08:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4179 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[27571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[27571](info): set up 2 sockets
Oct 25 20:08:59 dist-test-slave-d2pk krb5kdc[27571](info): commencing operation
krb5kdc: starting...
W20251025 20:09:01.606247 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.917s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:09:01 dist-test-slave-d2pk krb5kdc[27571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:09:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:01.734413 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:37377
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:01.839998 27587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:01.840209 27587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:01.840309 27587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:01.843411 27587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:01.843544 27587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:01.843642 27587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:01.843685 27587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:01.843724 27587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:01.848006 27587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:01.849013 27587 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:01.849866 27587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:01.855096 27592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:01.855113 27593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:01.855147 27595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:01.855244 27587 server_base.cc:1047] running on GCE node
I20251025 20:09:01.855826 27587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:01.856637 27587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:01.857774 27587 hybrid_clock.cc:648] HybridClock initialized: now 1761422941857759 us; error 30 us; skew 500 ppm
W20251025 20:09:01.860719 27587 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-25T20:09:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2193 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:09:01 dist-test-slave-d2pk krb5kdc[27600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:09:01 dist-test-slave-d2pk krb5kdc[27600](info): set up 2 sockets
Oct 25 20:09:01 dist-test-slave-d2pk krb5kdc[27600](info): commencing operation
krb5kdc: starting...
W20251025 20:09:03.778348 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.895s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:09:03 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:09:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:03.897866 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43255
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:03.996984 27616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:03.997167 27616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:03.997242 27616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:04.000304 27616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:04.000358 27616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:04.000394 27616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:04.000427 27616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:04.000468 27616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:04.004261 27616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:04.005290 27616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:04.006052 27616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:04.011083 27624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.011184 27622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.011247 27616 server_base.cc:1047] running on GCE node
W20251025 20:09:04.011088 27621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.011758 27616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:04.012537 27616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:04.013684 27616 hybrid_clock.cc:648] HybridClock initialized: now 1761422944013669 us; error 32 us; skew 500 ppm
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:04.016096 27616 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:09:04.017045 27616 webserver.cc:492] Webserver started at http://127.18.82.126:44961/ using document root <none> and password file <none>
I20251025 20:09:04.017621 27616 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:04.017675 27616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:04.017835 27616 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:04.019362 27616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "dbf497fbf4084a5293b43348943b762a"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "2b87b1174b0fdef25d123e843c0d8e4f"
server_key_iv: "40bf929c94df439f9cf90fd86c22a275"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.019840 27616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "dbf497fbf4084a5293b43348943b762a"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "2b87b1174b0fdef25d123e843c0d8e4f"
server_key_iv: "40bf929c94df439f9cf90fd86c22a275"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.022832 27616 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:09:04.024823 27631 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.025755 27616 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:04.025852 27616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "dbf497fbf4084a5293b43348943b762a"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "2b87b1174b0fdef25d123e843c0d8e4f"
server_key_iv: "40bf929c94df439f9cf90fd86c22a275"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.025931 27616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:04.040936 27616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:04.048287 27616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:04.048487 27616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:04.055328 27616 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:43255
I20251025 20:09:04.055330 27683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:43255 every 8 connection(s)
I20251025 20:09:04.056224 27616 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:09:04.058779 27684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:04.062620 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27616
I20251025 20:09:04.062757 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:09:04.062997 18761 external_mini_cluster.cc:1468] Setting key 01ad9b3d6125f4d8773814ae1627a465
I20251025 20:09:04.064349 27684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Bootstrap starting.
I20251025 20:09:04.066707 27684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:04.067366 27684 log.cc:826] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:04.069269 27684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: No bootstrap required, opened a new log
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:04.071789 27684 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:04.071964 27684 raft_consensus.cc:385] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:04.072023 27684 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf497fbf4084a5293b43348943b762a, State: Initialized, Role: FOLLOWER
I20251025 20:09:04.072440 27684 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:04.072607 27684 raft_consensus.cc:399] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:09:04.072687 27684 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:09:04.072777 27684 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:04.073758 27684 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:04.074117 27684 leader_election.cc:304] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbf497fbf4084a5293b43348943b762a; no voters:
I20251025 20:09:04.074450 27684 leader_election.cc:290] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:09:04.074762 27689 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:09:04.074872 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.064132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.18.82.126:43255)
1025 20:09:04.064489 (+ 357us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:04.064498 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:04.064530 (+ 32us) server_negotiation.cc:408] Connection header received
1025 20:09:04.065015 (+ 485us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:04.065030 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:04.065293 (+ 263us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:04.065587 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:04.066598 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.067348 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.068083 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.068283 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.070609 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:04.070626 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:04.070636 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:04.070665 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:04.072241 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.072700 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.072706 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.072712 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.072806 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.073040 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.073043 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.073045 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.073458 (+ 413us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:04.073717 (+ 259us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:04.073887 (+ 170us) server_negotiation.cc:300] Negotiation successful
1025 20:09:04.074125 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20251025 20:09:04.074944 27689 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 LEADER]: Becoming Leader. State: Replica: dbf497fbf4084a5293b43348943b762a, State: Running, Role: LEADER
I20251025 20:09:04.075232 27689 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:04.075480 27684 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:09:04.076707 27690 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbf497fbf4084a5293b43348943b762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } } }
I20251025 20:09:04.076807 27690 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:04.077049 27691 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbf497fbf4084a5293b43348943b762a. Latest consensus state: current_term: 1 leader_uuid: "dbf497fbf4084a5293b43348943b762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } } }
I20251025 20:09:04.077142 27691 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:04.077389 27698 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:09:04.080060 27698 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:09:04.084755 27698 catalog_manager.cc:1357] Generated new cluster ID: d168a9382a934dbbb05c76ef68dace53
I20251025 20:09:04.084817 27698 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:09:04.099244 27698 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:09:04.100304 27698 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:09:04.109450 27698 catalog_manager.cc:6022] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Generated new TSK 0
I20251025 20:09:04.110143 27698 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:04.156395 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43255
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:04.260736 27712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:04.260927 27712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:04.261013 27712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:04.263844 27712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:04.263897 27712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:04.263998 27712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:09:04.267905 27712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:04.268775 27712 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:04.269572 27712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:04.275542 27717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.275547 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.275667 27712 server_base.cc:1047] running on GCE node
W20251025 20:09:04.275548 27718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.276099 27712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:04.276577 27712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:04.277756 27712 hybrid_clock.cc:648] HybridClock initialized: now 1761422944277742 us; error 42 us; skew 500 ppm
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:04.280014 27712 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:09:04.280930 27712 webserver.cc:492] Webserver started at http://127.18.82.65:43407/ using document root <none> and password file <none>
I20251025 20:09:04.281431 27712 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:04.281481 27712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:04.281677 27712 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:04.283218 27712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "4ba4492922354d5d92e76f5d739b2be7"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "507dbcfaf353ac5846e50ee13c51dec9"
server_key_iv: "d763417111335752b930b89c61c51005"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.283667 27712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "4ba4492922354d5d92e76f5d739b2be7"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "507dbcfaf353ac5846e50ee13c51dec9"
server_key_iv: "d763417111335752b930b89c61c51005"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.286756 27712 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251025 20:09:04.288754 27727 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.289633 27712 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:04.289723 27712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "4ba4492922354d5d92e76f5d739b2be7"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "507dbcfaf353ac5846e50ee13c51dec9"
server_key_iv: "d763417111335752b930b89c61c51005"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.289795 27712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:04.304816 27712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:04.307142 27712 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:04.307293 27712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:04.307780 27712 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:04.308513 27712 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:04.308557 27712 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.308610 27712 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:04.308640 27712 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.316666 27712 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:41717
I20251025 20:09:04.316707 27840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:41717 every 8 connection(s)
I20251025 20:09:04.317476 27712 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:09:04.321203 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27712
I20251025 20:09:04.321312 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:09:04.321540 18761 external_mini_cluster.cc:1468] Setting key 7a5796d0d97986726ccf24cb167bf4e3
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:04.329151 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.319055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:41991 (local address 127.18.82.126:43255)
1025 20:09:04.319191 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:04.319196 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:04.319771 (+ 575us) server_negotiation.cc:408] Connection header received
1025 20:09:04.320469 (+ 698us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:04.320472 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:04.320518 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:04.320590 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:04.322129 (+ 1539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.322596 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.323174 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.323433 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.325588 (+ 2155us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:04.325606 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:04.325608 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:04.325633 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:04.327161 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.327785 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.327788 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.327789 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.327832 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.328149 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.328151 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.328152 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.328278 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:04.328347 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:04.328922 (+ 575us) server_negotiation.cc:300] Negotiation successful
1025 20:09:04.329020 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 20:09:04.330179 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.319268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:41991)
1025 20:09:04.319644 (+ 376us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:04.319675 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:04.320309 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:04.320708 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:04.320716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:04.321045 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:04.321990 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.322007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.322708 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.322711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:04.323069 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.323075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.323225 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.323785 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:04.323808 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:04.325411 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:04.327290 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.327296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.327308 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.327678 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.327917 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.327922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.327926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.328059 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.328368 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:04.328376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:04.328677 (+ 301us) client_negotiation.cc:769] Sending connection context
1025 20:09:04.328898 (+ 221us) client_negotiation.cc:240] Negotiation successful
1025 20:09:04.329314 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
I20251025 20:09:04.331303 27841 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43255
I20251025 20:09:04.331521 27841 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:04.332028 27841 heartbeater.cc:507] Master 127.18.82.126:43255 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
I20251025 20:09:04.333465 27648 ts_manager.cc:194] Registered new tserver with Master: 4ba4492922354d5d92e76f5d739b2be7 (127.18.82.65:41717)
I20251025 20:09:04.334640 27648 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:41991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:04.365922 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43255
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:04.463668 27848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:04.463840 27848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:04.463917 27848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:04.466949 27848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:04.467003 27848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:04.467109 27848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:09:04.470870 27848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:04.471735 27848 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:04.472451 27848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:04.478199 27853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.478207 27856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.478278 27854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.478361 27848 server_base.cc:1047] running on GCE node
I20251025 20:09:04.478734 27848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:04.479158 27848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:04.480306 27848 hybrid_clock.cc:648] HybridClock initialized: now 1761422944480298 us; error 23 us; skew 500 ppm
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:04.482576 27848 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:09:04.483477 27848 webserver.cc:492] Webserver started at http://127.18.82.66:40747/ using document root <none> and password file <none>
I20251025 20:09:04.483949 27848 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:04.484001 27848 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:04.484191 27848 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:04.485759 27848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "059445007c0a4543959a5d6b438e3eeb"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "df4150d76cfe06f14ad50a5e4212679b"
server_key_iv: "6aa798ac32601e8669d175bde18c9348"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.486174 27848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "059445007c0a4543959a5d6b438e3eeb"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "df4150d76cfe06f14ad50a5e4212679b"
server_key_iv: "6aa798ac32601e8669d175bde18c9348"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.489327 27848 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:09:04.491180 27863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.492019 27848 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:09:04.492105 27848 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "059445007c0a4543959a5d6b438e3eeb"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "df4150d76cfe06f14ad50a5e4212679b"
server_key_iv: "6aa798ac32601e8669d175bde18c9348"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.492182 27848 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:04.507622 27848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:04.509886 27848 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:04.510036 27848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:04.510514 27848 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:04.511277 27848 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:04.511327 27848 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.511372 27848 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:04.511402 27848 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.519668 27848 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:35305
I20251025 20:09:04.519687 27976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:35305 every 8 connection(s)
I20251025 20:09:04.520483 27848 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:04.530730 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27848
I20251025 20:09:04.530848 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:09:04.531118 18761 external_mini_cluster.cc:1468] Setting key f56b7afd46d42cdb60ff207468384db1
I20251025 20:09:04.531379 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.522008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:54559 (local address 127.18.82.126:43255)
1025 20:09:04.522137 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:04.522141 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:04.522709 (+ 568us) server_negotiation.cc:408] Connection header received
1025 20:09:04.523482 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:04.523486 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:04.523546 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:04.523623 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:04.525159 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.525846 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.526487 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.526685 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.528490 (+ 1805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:04.528507 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:04.528509 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:04.528532 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:04.529723 (+ 1191us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.530136 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.530138 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.530140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.530180 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.530407 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.530409 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.530410 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.530532 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:04.530598 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:04.531146 (+ 548us) server_negotiation.cc:300] Negotiation successful
1025 20:09:04.531248 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:09:04.531998 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.522212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:54559)
1025 20:09:04.522584 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:04.522615 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:04.523289 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:04.523775 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:04.523783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:04.524134 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:04.525011 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.525024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.525961 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.525965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:04.526371 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.526379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.526570 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.527068 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:04.527089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:04.528367 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:04.529817 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.529822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.529830 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.530058 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.530250 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.530253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.530254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.530340 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.530627 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:04.530636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:04.530897 (+ 261us) client_negotiation.cc:769] Sending connection context
1025 20:09:04.531110 (+ 213us) client_negotiation.cc:240] Negotiation successful
1025 20:09:04.531354 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
I20251025 20:09:04.532886 27977 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43255
I20251025 20:09:04.533082 27977 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:04.533499 27977 heartbeater.cc:507] Master 127.18.82.126:43255 requested a full tablet report, sending...
I20251025 20:09:04.534451 27648 ts_manager.cc:194] Registered new tserver with Master: 059445007c0a4543959a5d6b438e3eeb (127.18.82.66:35305)
I20251025 20:09:04.535243 27648 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:54559
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:04.572919 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:43255
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:04.673697 27984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:04.673872 27984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:04.673942 27984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:04.676978 27984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:04.677042 27984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:04.677132 27984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:09:04.680994 27984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:04.681913 27984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:04.682637 27984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:04.688494 27989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.688596 27984 server_base.cc:1047] running on GCE node
W20251025 20:09:04.688490 27990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.688500 27992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.689008 27984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:04.689496 27984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:04.690686 27984 hybrid_clock.cc:648] HybridClock initialized: now 1761422944690661 us; error 43 us; skew 500 ppm
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:04.693737 27984 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:09:04.694905 27984 webserver.cc:492] Webserver started at http://127.18.82.67:43593/ using document root <none> and password file <none>
I20251025 20:09:04.695549 27984 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:04.695617 27984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:04.695842 27984 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:04.698006 27984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "691a11ab932c4b2bb4ceb6a3da54d07e"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "87c7ab757a14a768b93f357c847bb326"
server_key_iv: "c19c1af63ea8e5036fce02e38467e2ab"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.698494 27984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "691a11ab932c4b2bb4ceb6a3da54d07e"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "87c7ab757a14a768b93f357c847bb326"
server_key_iv: "c19c1af63ea8e5036fce02e38467e2ab"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.701488 27984 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:09:04.703428 27999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.704325 27984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:04.704421 27984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "691a11ab932c4b2bb4ceb6a3da54d07e"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "87c7ab757a14a768b93f357c847bb326"
server_key_iv: "c19c1af63ea8e5036fce02e38467e2ab"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.704491 27984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:04.718868 27984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:04.721050 27984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:04.721175 27984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:04.721624 27984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:04.722378 27984 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:04.722419 27984 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.722463 27984 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:04.722483 27984 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.730971 27984 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34921
I20251025 20:09:04.731043 28112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34921 every 8 connection(s)
I20251025 20:09:04.732077 27984 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:09:04.737709 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 27984
I20251025 20:09:04.737840 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:09:04.738118 18761 external_mini_cluster.cc:1468] Setting key aded815f503e8d4293151f56ae51990c
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:04.744738 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.735334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:39133 (local address 127.18.82.126:43255)
1025 20:09:04.735473 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:04.735478 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:04.735494 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:09:04.735967 (+ 473us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:04.735972 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:04.736038 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:04.736133 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:04.738419 (+ 2286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.739015 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.739780 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.739911 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.742148 (+ 2237us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:04.742163 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:04.742166 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:04.742189 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:04.743341 (+ 1152us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.743750 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.743753 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.743754 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.743786 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:04.743979 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:04.743981 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:04.743983 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:04.744107 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:04.744170 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:04.744540 (+ 370us) server_negotiation.cc:300] Negotiation successful
1025 20:09:04.744626 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:09:04.745296 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:04.734317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:39133)
1025 20:09:04.734831 (+ 514us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:04.734876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:04.735743 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:04.736291 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:04.736300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:04.736823 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:04.738084 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.738102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.739138 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:04.739141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:04.739676 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:04.739687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:04.739892 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:04.740563 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:04.740589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:04.742024 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:04.743442 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.743446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.743455 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.743678 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.743846 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:04.743848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:04.743850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:04.743921 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:04.744177 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:04.744181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:04.744368 (+ 187us) client_negotiation.cc:769] Sending connection context
1025 20:09:04.744518 (+ 150us) client_negotiation.cc:240] Negotiation successful
1025 20:09:04.744711 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20251025 20:09:04.746176 28113 heartbeater.cc:344] Connected to a master server at 127.18.82.126:43255
I20251025 20:09:04.746390 28113 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:04.746805 28113 heartbeater.cc:507] Master 127.18.82.126:43255 requested a full tablet report, sending...
I20251025 20:09:04.747655 27648 ts_manager.cc:194] Registered new tserver with Master: 691a11ab932c4b2bb4ceb6a3da54d07e (127.18.82.67:34921)
I20251025 20:09:04.748129 27648 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:39133
I20251025 20:09:04.751503 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.65.keytab.
Entry for principal oryx/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.65.keytab.
WARNING: no policy specified for oryx/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.66.keytab.
Entry for principal oryx/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.66.keytab.
WARNING: no policy specified for oryx/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.67.keytab.
Entry for principal oryx/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.67.keytab.
I20251025 20:09:04.812083 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27712
I20251025 20:09:04.818405 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27848
I20251025 20:09:04.824234 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27984
I20251025 20:09:04.829571 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 27616
I20251025 20:09:04.835733 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43255
--webserver_interface=127.18.82.126
--webserver_port=44961
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:04.933805 28122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:04.933983 28122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:04.934054 28122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:04.936980 28122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:04.937026 28122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:04.937047 28122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:04.937064 28122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:04.937080 28122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:04.940917 28122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=44961
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:04.941817 28122 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:04.942548 28122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:04.947575 28128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.947639 28127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:04.947607 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:04.947821 28122 server_base.cc:1047] running on GCE node
I20251025 20:09:04.948225 28122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:04.949044 28122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:04.950174 28122 hybrid_clock.cc:648] HybridClock initialized: now 1761422944950158 us; error 32 us; skew 500 ppm
Oct 25 20:09:04 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:04.952522 28122 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:09:04.953428 28122 webserver.cc:492] Webserver started at http://127.18.82.126:44961/ using document root <none> and password file <none>
I20251025 20:09:04.953925 28122 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:04.953987 28122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:04.957207 28122 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:09:04.958901 28137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:04.959736 28122 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 20:09:04.959830 28122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "dbf497fbf4084a5293b43348943b762a"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "2b87b1174b0fdef25d123e843c0d8e4f"
server_key_iv: "40bf929c94df439f9cf90fd86c22a275"
server_key_version: "encryptionkey@0"
I20251025 20:09:04.960177 28122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:04.989086 28122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:04.991408 28122 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:04.991569 28122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:04.998253 28189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:43255 every 8 connection(s)
I20251025 20:09:04.998266 28122 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:43255
I20251025 20:09:04.999423 28122 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:09:05.001411 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28122
I20251025 20:09:05.001924 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:41717
--local_ip_for_outbound_sockets=127.18.82.65
--tserver_master_addrs=127.18.82.126:43255
--webserver_port=43407
--webserver_interface=127.18.82.65
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.82.65
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 20:09:05.004253 28190 sys_catalog.cc:263] Verifying existing consensus state
I20251025 20:09:05.005780 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Bootstrap starting.
I20251025 20:09:05.021109 28190 log.cc:826] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:05.026883 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 20:09:05.027207 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Bootstrap complete.
I20251025 20:09:05.030534 28190 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:05.030851 28190 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf497fbf4084a5293b43348943b762a, State: Initialized, Role: FOLLOWER
I20251025 20:09:05.031265 28190 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:05.031368 28190 raft_consensus.cc:399] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:09:05.031425 28190 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:09:05.031508 28190 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 1 FOLLOWER]: Advancing to term 2
I20251025 20:09:05.032884 28190 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:05.033166 28190 leader_election.cc:304] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbf497fbf4084a5293b43348943b762a; no voters:
I20251025 20:09:05.033406 28190 leader_election.cc:290] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [CANDIDATE]: Term 2 election: Requested vote from peers
I20251025 20:09:05.033494 28194 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 2 FOLLOWER]: Leader election won for term 2
I20251025 20:09:05.033710 28194 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [term 2 LEADER]: Becoming Leader. State: Replica: dbf497fbf4084a5293b43348943b762a, State: Running, Role: LEADER
I20251025 20:09:05.033977 28194 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } }
I20251025 20:09:05.034554 28190 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:09:05.034849 28195 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbf497fbf4084a5293b43348943b762a. Latest consensus state: current_term: 2 leader_uuid: "dbf497fbf4084a5293b43348943b762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } } }
I20251025 20:09:05.035115 28195 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:05.035033 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dbf497fbf4084a5293b43348943b762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf497fbf4084a5293b43348943b762a" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 43255 } } }
I20251025 20:09:05.035457 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:05.036113 28199 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:09:05.038949 28199 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:09:05.039795 28199 catalog_manager.cc:1269] Loaded cluster ID: d168a9382a934dbbb05c76ef68dace53
I20251025 20:09:05.039853 28199 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:09:05.042177 28199 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:09:05.042833 28199 catalog_manager.cc:6033] T 00000000000000000000000000000000 P dbf497fbf4084a5293b43348943b762a: Loaded TSK: 0
I20251025 20:09:05.043953 28199 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251025 20:09:05.105679 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:05.105842 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:05.105894 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:05.108983 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:05.109031 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:05.109120 28192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:09:05.113008 28192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.65.keytab
--principal=oryx/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:41717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=43407
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:05.113914 28192 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:05.114693 28192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:05.120667 28217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:05.120671 28216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:05.120829 28192 server_base.cc:1047] running on GCE node
W20251025 20:09:05.120898 28219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:05.121207 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:05.121734 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:05.122895 28192 hybrid_clock.cc:648] HybridClock initialized: now 1761422945122878 us; error 27 us; skew 500 ppm
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422945, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:05.125167 28192 init.cc:377] Logged in from keytab as oryx/127.18.82.65@KRBTEST.COM (short username oryx)
I20251025 20:09:05.126178 28192 webserver.cc:492] Webserver started at http://127.18.82.65:43407/ using document root <none> and password file <none>
I20251025 20:09:05.126744 28192 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:05.126813 28192 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:05.129933 28192 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:09:05.131672 28226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.132655 28192 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:05.132766 28192 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "4ba4492922354d5d92e76f5d739b2be7"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "507dbcfaf353ac5846e50ee13c51dec9"
server_key_iv: "d763417111335752b930b89c61c51005"
server_key_version: "encryptionkey@0"
I20251025 20:09:05.133160 28192 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:05.158697 28192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:05.161774 28192 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:05.161947 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:05.162492 28192 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:05.163369 28192 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:05.163417 28192 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.163471 28192 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:05.163503 28192 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.174273 28192 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:41717
I20251025 20:09:05.174353 28339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:41717 every 8 connection(s)
I20251025 20:09:05.175244 28192 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:09:05.177467 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28192
I20251025 20:09:05.177989 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:35305
--local_ip_for_outbound_sockets=127.18.82.66
--tserver_master_addrs=127.18.82.126:43255
--webserver_port=40747
--webserver_interface=127.18.82.66
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.82.66
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.188441 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.177097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:47341)
1025 20:09:05.177815 (+ 718us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.177857 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.178622 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.179841 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.179853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.180433 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.181753 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.181777 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.183503 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.183507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.184160 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.184169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.184380 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.184990 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.185020 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.187723 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":129,"threads_started":1}
W20251025 20:09:05.188621 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.177086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:47341 (local address 127.18.82.126:43255)
1025 20:09:05.178895 (+ 1809us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.178904 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.178933 (+ 29us) server_negotiation.cc:408] Connection header received
1025 20:09:05.179144 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.179153 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.179411 (+ 258us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.179738 (+ 327us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.181969 (+ 2231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.183331 (+ 1362us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.184288 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.184619 (+ 331us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.187942 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:47341: BlockingRecv error: recv got EOF from 127.18.82.65:47341 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":127,"threads_started":1}
W20251025 20:09:05.189198 28340 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:43255 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.82.126:43255: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.194604 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.189722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:53869)
1025 20:09:05.189899 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.189913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.189998 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.190271 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.190274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.190428 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.190666 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.190674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.191692 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.191696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.192140 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.192149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.192247 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.192720 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.192735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.194516 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20251025 20:09:05.194749 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.189792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:53869 (local address 127.18.82.126:43255)
1025 20:09:05.189914 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.189918 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.189959 (+ 41us) server_negotiation.cc:408] Connection header received
1025 20:09:05.190088 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.190094 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.190150 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.190242 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.190785 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.191582 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.192263 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.192478 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.194657 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:53869: BlockingRecv error: recv got EOF from 127.18.82.65:53869 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.199072 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.195014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:49537)
1025 20:09:05.195267 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.195278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.195381 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.195593 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.195596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.195725 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.195853 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.195857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.196455 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.196458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.196767 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.196773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.196865 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.197365 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.197376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.198936 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
W20251025 20:09:05.199251 28340 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251025 20:09:05.199337 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.195119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:49537 (local address 127.18.82.126:43255)
1025 20:09:05.195214 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.195218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.195339 (+ 121us) server_negotiation.cc:408] Connection header received
1025 20:09:05.195456 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.195461 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.195501 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.195563 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.195952 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.196362 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.196862 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.197041 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.199063 (+ 2022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:49537: BlockingRecv error: recv got EOF from 127.18.82.65:49537 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251025 20:09:05.284053 28344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:05.284255 28344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:05.284336 28344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:05.287386 28344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:05.287490 28344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:05.287609 28344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:09:05.291239 28344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.66.keytab
--principal=oryx/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:35305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=40747
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:05.292074 28344 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:05.292819 28344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:05.298936 28350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:05.298942 28351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:05.298985 28344 server_base.cc:1047] running on GCE node
W20251025 20:09:05.298942 28353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:05.299474 28344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:05.299963 28344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:05.301098 28344 hybrid_clock.cc:648] HybridClock initialized: now 1761422945301089 us; error 26 us; skew 500 ppm
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422945, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:05.303375 28344 init.cc:377] Logged in from keytab as oryx/127.18.82.66@KRBTEST.COM (short username oryx)
I20251025 20:09:05.304291 28344 webserver.cc:492] Webserver started at http://127.18.82.66:40747/ using document root <none> and password file <none>
I20251025 20:09:05.304790 28344 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:05.304849 28344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:05.308060 28344 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:05.309610 28360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.310438 28344 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:05.310523 28344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "059445007c0a4543959a5d6b438e3eeb"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "df4150d76cfe06f14ad50a5e4212679b"
server_key_iv: "6aa798ac32601e8669d175bde18c9348"
server_key_version: "encryptionkey@0"
I20251025 20:09:05.310879 28344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:05.327340 28344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:05.329550 28344 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:05.329694 28344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:05.330155 28344 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:05.330917 28344 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:05.330961 28344 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.331002 28344 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:05.331032 28344 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.339591 28344 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:35305
I20251025 20:09:05.339620 28473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:35305 every 8 connection(s)
I20251025 20:09:05.340482 28344 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:09:05.343778 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28344
I20251025 20:09:05.344309 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:34921
--local_ip_for_outbound_sockets=127.18.82.67
--tserver_master_addrs=127.18.82.126:43255
--webserver_port=43593
--webserver_interface=127.18.82.67
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.82.67
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
W20251025 20:09:05.350790 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.342053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:42693 (local address 127.18.82.126:43255)
1025 20:09:05.342205 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.342210 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.342819 (+ 609us) server_negotiation.cc:408] Connection header received
1025 20:09:05.343556 (+ 737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.343559 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.343610 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.343675 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.345826 (+ 2151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.346675 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.347363 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.347568 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.350695 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:42693: BlockingRecv error: recv got EOF from 127.18.82.66:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20251025 20:09:05.351382 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.342305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:42693)
1025 20:09:05.342698 (+ 393us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.342731 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.343387 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.343869 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.343882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.344392 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.345668 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.345688 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.346806 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.346811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.347234 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.347243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.347447 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.348085 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.348133 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.350599 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
W20251025 20:09:05.351845 28474 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:43255 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.82.126:43255: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.356716 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.352223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:49997)
1025 20:09:05.352466 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.352478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.352563 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.352781 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.352786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.352933 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.353161 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.353171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.353815 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.353821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.354389 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.354399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.354525 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.354995 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.355011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.356592 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165}
W20251025 20:09:05.356805 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.352315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:49997 (local address 127.18.82.126:43255)
1025 20:09:05.352478 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.352482 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.352535 (+ 53us) server_negotiation.cc:408] Connection header received
1025 20:09:05.352653 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.352655 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.352691 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.352749 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.353258 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.353716 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.354514 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.354638 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.356709 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:49997: BlockingRecv error: recv got EOF from 127.18.82.66:49997 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.360991 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.357186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:35965)
1025 20:09:05.357291 (+ 105us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.357303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.357360 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.357617 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.357619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.357727 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.357859 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.357863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.358713 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.358716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.359009 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.359015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.359089 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.359469 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.359481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.360924 (+ 1443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20251025 20:09:05.361124 28474 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251025 20:09:05.361251 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.357262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:35965 (local address 127.18.82.126:43255)
1025 20:09:05.357358 (+ 96us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.357362 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.357376 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:09:05.357417 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.357422 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.357473 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.357613 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.357978 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.358622 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.359102 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.359261 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.361041 (+ 1780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:35965: BlockingRecv error: recv got EOF from 127.18.82.66:35965 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20251025 20:09:05.447845 28477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:05.448038 28477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:05.448094 28477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:05.451208 28477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:05.451258 28477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:05.451354 28477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:09:05.455422 28477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/krb5kdc/oryx_127.18.82.67.keytab
--principal=oryx/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:34921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=43593
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:05.456327 28477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:05.457108 28477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:05.463083 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:05.463094 28485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:05.463068 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:05.463325 28477 server_base.cc:1047] running on GCE node
I20251025 20:09:05.463711 28477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:05.464206 28477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:05.465376 28477 hybrid_clock.cc:648] HybridClock initialized: now 1761422945465347 us; error 39 us; skew 500 ppm
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422945, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:05.467779 28477 init.cc:377] Logged in from keytab as oryx/127.18.82.67@KRBTEST.COM (short username oryx)
I20251025 20:09:05.468693 28477 webserver.cc:492] Webserver started at http://127.18.82.67:43593/ using document root <none> and password file <none>
I20251025 20:09:05.469182 28477 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:05.469250 28477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:05.472502 28477 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20251025 20:09:05.474200 28492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.475071 28477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251025 20:09:05.475168 28477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "691a11ab932c4b2bb4ceb6a3da54d07e"
format_stamp: "Formatted at 2025-10-25 20:09:04 on dist-test-slave-d2pk"
server_key: "87c7ab757a14a768b93f357c847bb326"
server_key_iv: "c19c1af63ea8e5036fce02e38467e2ab"
server_key_version: "encryptionkey@0"
I20251025 20:09:05.475548 28477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:05.489244 28477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:05.491812 28477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:05.491964 28477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:05.492425 28477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:05.493175 28477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:05.493222 28477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.493278 28477 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:05.493305 28477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:05.501600 28477 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34921
I20251025 20:09:05.501628 28605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34921 every 8 connection(s)
I20251025 20:09:05.502460 28477 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.509902 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28477
W20251025 20:09:05.510761 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.503890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:57503 (local address 127.18.82.126:43255)
1025 20:09:05.504042 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.504047 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.504678 (+ 631us) server_negotiation.cc:408] Connection header received
1025 20:09:05.505398 (+ 720us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.505401 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.505448 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.505569 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.507019 (+ 1450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.507485 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.508137 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.508262 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.510649 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:57503: BlockingRecv error: recv got EOF from 127.18.82.67:57503 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:09:05.511101 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.504152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:57503)
1025 20:09:05.504541 (+ 389us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.504572 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.505217 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.505688 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.505695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.506019 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.506897 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.506911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.507579 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.507582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.508037 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.508045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.508192 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.508612 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.508630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.510533 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
W20251025 20:09:05.511432 28606 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:43255 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.82.126:43255: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.516000 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.511855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:60387)
1025 20:09:05.512048 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.512059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.512116 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.512310 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.512315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.512475 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.512690 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.512697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.513377 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.513380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.513766 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.513774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.513868 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.514365 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.514382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.515888 (+ 1506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20251025 20:09:05.516042 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.511949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:60387 (local address 127.18.82.126:43255)
1025 20:09:05.512064 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.512068 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.512087 (+ 19us) server_negotiation.cc:408] Connection header received
1025 20:09:05.512188 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.512192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.512228 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.512302 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.512785 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.513277 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.513874 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.513991 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.515975 (+ 1984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:60387: BlockingRecv error: recv got EOF from 127.18.82.67:60387 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20251025 20:09:05.518472 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.510642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.18.82.126:43255)
1025 20:09:05.510866 (+ 224us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.510870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.511121 (+ 251us) server_negotiation.cc:408] Connection header received
1025 20:09:05.511223 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.511226 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.511268 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.511349 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.512361 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.513017 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.513765 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.514415 (+ 650us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.514906 (+ 491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:05.514922 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:05.514931 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:05.514966 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:05.516863 (+ 1897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:05.517248 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:05.517252 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:05.517255 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:05.517330 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:05.517534 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:05.517553 (+ 19us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:05.517555 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:05.517841 (+ 286us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:05.518001 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:05.518184 (+ 183us) server_negotiation.cc:300] Negotiation successful
1025 20:09:05.518375 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:05 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:05.520373 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:05.516419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:32999)
1025 20:09:05.516609 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:05.516619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:05.516691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:05.516913 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:05.516916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:05.517067 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:05.517242 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.517249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.518016 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.518019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:05.518427 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:05.518435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.518519 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.518903 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:05.518917 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:05.520305 (+ 1388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20251025 20:09:05.520481 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:05.516473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:32999 (local address 127.18.82.126:43255)
1025 20:09:05.516578 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:05.516581 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:05.516683 (+ 102us) server_negotiation.cc:408] Connection header received
1025 20:09:05.516779 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:05.516782 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:05.516827 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:05.516910 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:05.517324 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.517919 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:05.518526 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:05.518680 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:05.520411 (+ 1731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:32999: BlockingRecv error: recv got EOF from 127.18.82.67:32999 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20251025 20:09:05.520511 28606 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:06.204113 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:06.199986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:44843)
1025 20:09:06.200260 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:06.200273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:06.200362 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:06.200584 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:06.200586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:06.200738 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:06.200936 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.200941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.201594 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.201597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:06.201946 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.201952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.202033 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.202513 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:06.202525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:06.204008 (+ 1483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":66,"threads_started":1}
W20251025 20:09:06.204243 28612 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:06.200057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:44843 (local address 127.18.82.126:43255)
1025 20:09:06.200305 (+ 248us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:06.200309 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:06.200320 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:09:06.200442 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:06.200445 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:06.200486 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:06.200553 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:06.201025 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.201492 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.202030 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.202151 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.204122 (+ 1971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:44843: BlockingRecv error: recv got EOF from 127.18.82.65:44843 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:06.366173 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:06.361719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:58251)
1025 20:09:06.361968 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:06.361982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:06.362079 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:06.362287 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:06.362290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:06.362466 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:06.362654 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.362659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.363301 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.363304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:06.363686 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.363697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.363788 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.364405 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:06.364418 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:06.366031 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
W20251025 20:09:06.366248 28612 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:06.361891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:58251 (local address 127.18.82.126:43255)
1025 20:09:06.362017 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:06.362023 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:06.362036 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:09:06.362161 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:06.362164 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:06.362205 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:06.362262 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:06.362756 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.363211 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.363776 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.363980 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.366146 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:58251: BlockingRecv error: recv got EOF from 127.18.82.66:58251 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:06 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:06.525718 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:06.521224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:46119)
1025 20:09:06.521483 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:06.521496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:06.521623 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:06.521873 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:06.521876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:06.522044 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:06.522286 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.522299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.522952 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.522955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:06.523317 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:06.523323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.523427 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.523934 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:06.523946 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:06.525571 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
W20251025 20:09:06.525753 28612 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:06.521297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:46119 (local address 127.18.82.126:43255)
1025 20:09:06.521446 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:06.521451 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:06.521599 (+ 148us) server_negotiation.cc:408] Connection header received
1025 20:09:06.521711 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:06.521716 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:06.521766 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:06.521836 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:06.522409 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.522856 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:06.523428 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:06.523615 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:06.525667 (+ 2052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:46119: BlockingRecv error: recv got EOF from 127.18.82.67:46119 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:07.209427 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:07.204834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:35651)
1025 20:09:07.205124 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:07.205138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:07.205237 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:07.205510 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:07.205512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:07.205726 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:07.205910 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.205916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.206603 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.206606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:07.206986 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.206993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.207080 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.207616 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:07.207628 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:07.209289 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
W20251025 20:09:07.209671 28616 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:07.204954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:35651 (local address 127.18.82.126:43255)
1025 20:09:07.205170 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:07.205174 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:07.205187 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:09:07.205346 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:07.205349 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:07.205393 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:07.205468 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:07.205996 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.206499 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.207082 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.207211 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.209434 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:35651: BlockingRecv error: recv got EOF from 127.18.82.65:35651 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:07.371490 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:07.366960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:47945)
1025 20:09:07.367213 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:07.367225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:07.367326 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:07.367563 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:07.367565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:07.367733 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:07.367957 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.367966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.368835 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.368839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:07.369205 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.369212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.369295 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.369845 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:07.369858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:07.371390 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
W20251025 20:09:07.371628 28616 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:07.367005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:47945 (local address 127.18.82.126:43255)
1025 20:09:07.367148 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:07.367152 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:07.367295 (+ 143us) server_negotiation.cc:408] Connection header received
1025 20:09:07.367401 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:07.367404 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:07.367450 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:07.367514 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:07.368076 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.368730 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.369297 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.369434 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.371497 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:47945: BlockingRecv error: recv got EOF from 127.18.82.66:47945 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:07 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:07.530642 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:07.526422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:51745)
1025 20:09:07.526677 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:07.526708 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:07.526802 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:07.527019 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:07.527021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:07.527194 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:07.527407 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.527413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.528029 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.528031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:07.528366 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:07.528371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.528445 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.528904 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:07.528915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:07.530510 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
W20251025 20:09:07.530740 28616 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:07.526548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:51745 (local address 127.18.82.126:43255)
1025 20:09:07.526702 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:07.526706 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:07.526771 (+ 65us) server_negotiation.cc:408] Connection header received
1025 20:09:07.526876 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:07.526878 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:07.526923 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:07.526988 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:07.527510 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.527958 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:07.528443 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:07.528551 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:07.530640 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:51745: BlockingRecv error: recv got EOF from 127.18.82.67:51745 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:08.214529 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:08.210123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:48053)
1025 20:09:08.210384 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:08.210395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:08.210477 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:08.210691 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:08.210694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:08.210855 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:08.211048 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.211054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.211660 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.211662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:08.212048 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.212056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.212160 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.212738 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:08.212750 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:08.214356 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
W20251025 20:09:08.214610 28620 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:08.210286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:48053 (local address 127.18.82.126:43255)
1025 20:09:08.210490 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:08.210493 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:08.210504 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:09:08.210534 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:08.210536 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:08.210574 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:08.210661 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:08.211137 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.211581 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.212141 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.212308 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.214514 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:48053: BlockingRecv error: recv got EOF from 127.18.82.65:48053 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:08.376394 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:08.372065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:50251)
1025 20:09:08.372329 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:08.372345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:08.372453 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:08.372683 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:08.372686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:08.372850 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:08.373017 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.373022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.373727 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.373732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:08.374068 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.374074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.374173 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.374697 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:08.374710 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:08.376283 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
W20251025 20:09:08.376509 28620 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:08.372224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:50251 (local address 127.18.82.126:43255)
1025 20:09:08.372355 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:08.372361 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:08.372398 (+ 37us) server_negotiation.cc:408] Connection header received
1025 20:09:08.372540 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:08.372543 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:08.372583 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:08.372645 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:08.373129 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.373633 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.374173 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.374374 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.376400 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:50251: BlockingRecv error: recv got EOF from 127.18.82.66:50251 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:08 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:08.535566 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:08.531326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:38723)
1025 20:09:08.531569 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:08.531581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:08.531670 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:08.531868 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:08.531870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:08.532017 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:08.532199 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.532204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.532821 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.532823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:08.533151 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:08.533156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.533231 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.533764 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:08.533777 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:08.535455 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
W20251025 20:09:08.535679 28620 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:08.531396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:38723 (local address 127.18.82.126:43255)
1025 20:09:08.531536 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:08.531540 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:08.531642 (+ 102us) server_negotiation.cc:408] Connection header received
1025 20:09:08.531741 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:08.531744 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:08.531787 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:08.531846 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:08.532308 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.532750 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:08.533221 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:08.533324 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:08.535573 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:38723: BlockingRecv error: recv got EOF from 127.18.82.67:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:09.222923 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:09.215270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:38855)
1025 20:09:09.215612 (+ 342us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:09.215708 (+ 96us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:09.215859 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:09.216096 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:09.216098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:09.216259 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:09.216438 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.216443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.217058 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.217060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:09.217392 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.217397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.217470 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.221154 (+ 3684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:09.221167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:09.222817 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
W20251025 20:09:09.223094 28624 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:09.215340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:38855 (local address 127.18.82.126:43255)
1025 20:09:09.215637 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:09.215641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:09.215778 (+ 137us) server_negotiation.cc:408] Connection header received
1025 20:09:09.215950 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:09.215952 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:09.215997 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:09.216066 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:09.216529 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.216984 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.217471 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.217630 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.222940 (+ 5310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:38855: BlockingRecv error: recv got EOF from 127.18.82.65:38855 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:09.381469 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:09.377085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:51043)
1025 20:09:09.377324 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:09.377336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:09.377424 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:09.377664 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:09.377667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:09.377819 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:09.377996 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.378001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.378599 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.378601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:09.378931 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.378936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.379012 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.379770 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:09.379784 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:09.381318 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
W20251025 20:09:09.381546 28624 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:09.377141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:51043 (local address 127.18.82.126:43255)
1025 20:09:09.377254 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:09.377258 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:09.377391 (+ 133us) server_negotiation.cc:408] Connection header received
1025 20:09:09.377490 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:09.377492 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:09.377563 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:09.377633 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:09.378078 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.378522 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.379000 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.379130 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.381427 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:51043: BlockingRecv error: recv got EOF from 127.18.82.66:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:09 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:09.540514 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:09.536254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:60613)
1025 20:09:09.536490 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:09.536501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:09.536608 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:09.536835 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:09.536837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:09.537004 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:09.537214 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.537220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.537928 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.537932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:09.538268 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:09.538274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.538381 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.538878 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:09.538891 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:09.540402 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
W20251025 20:09:09.540635 28624 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:09.536287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:60613 (local address 127.18.82.126:43255)
1025 20:09:09.536409 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:09.536414 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:09.536560 (+ 146us) server_negotiation.cc:408] Connection header received
1025 20:09:09.536688 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:09.536691 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:09.536735 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:09.536790 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:09.537335 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.537824 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:09.538370 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:09.538557 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:09.540525 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:60613: BlockingRecv error: recv got EOF from 127.18.82.67:60613 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:10.228137 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:10.223699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:46315)
1025 20:09:10.223937 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:10.223948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:10.224023 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:10.224232 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:10.224234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:10.224407 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:10.224577 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.224593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.225281 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.225284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:10.225720 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.225728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.225817 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.226332 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:10.226349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:10.228007 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20251025 20:09:10.228255 28628 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:10.223868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:46315 (local address 127.18.82.126:43255)
1025 20:09:10.224060 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:10.224064 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:10.224074 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:09:10.224103 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:10.224105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:10.224142 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:10.224211 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:10.224710 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.225204 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.225806 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.225928 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.228134 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:46315: BlockingRecv error: recv got EOF from 127.18.82.65:46315 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:10.387085 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:10.382334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:52915)
1025 20:09:10.382586 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:10.382602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:10.382712 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:10.382946 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:10.382949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:10.383140 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:10.383312 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.383317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.383980 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.383983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:10.384321 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.384327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.384425 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.385028 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:10.385041 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:10.386944 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
W20251025 20:09:10.387172 28628 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:10.382482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:52915 (local address 127.18.82.126:43255)
1025 20:09:10.382617 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:10.382625 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:10.382676 (+ 51us) server_negotiation.cc:408] Connection header received
1025 20:09:10.382799 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:10.382803 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:10.382848 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:10.382910 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:10.383429 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.383879 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.384420 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.384615 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.387065 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:52915: BlockingRecv error: recv got EOF from 127.18.82.66:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:10 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:10.545823 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:10.541428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:58899)
1025 20:09:10.541697 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:10.541713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:10.541821 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:10.542049 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:10.542052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:10.542239 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:10.542414 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.542420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.543079 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.543082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:10.543488 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:10.543496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.543590 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.544115 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:10.544126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:10.545673 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
W20251025 20:09:10.545928 28628 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:10.541499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:58899 (local address 127.18.82.126:43255)
1025 20:09:10.541658 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:10.541663 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:10.541787 (+ 124us) server_negotiation.cc:408] Connection header received
1025 20:09:10.541908 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:10.541911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:10.541954 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:10.542013 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:10.542518 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.542974 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:10.543590 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:10.543771 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:10.545797 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:58899: BlockingRecv error: recv got EOF from 127.18.82.67:58899 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:11.233479 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:11.228987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:45705)
1025 20:09:11.229240 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:11.229252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:11.229350 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:11.229583 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:11.229586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:11.229745 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:11.229926 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.229931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.230767 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.230772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:11.231220 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.231228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.231337 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.231846 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:11.231859 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:11.233381 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
W20251025 20:09:11.233640 28632 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:11.229046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:45705 (local address 127.18.82.126:43255)
1025 20:09:11.229276 (+ 230us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:11.229279 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:11.229292 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:09:11.229429 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:11.229431 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:11.229472 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:11.229554 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:11.230023 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.230648 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.231339 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.231463 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.233493 (+ 2030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:45705: BlockingRecv error: recv got EOF from 127.18.82.65:45705 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:11.392323 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:11.387837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:33883)
1025 20:09:11.388087 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:11.388100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:11.388213 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:11.388494 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:11.388497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:11.388688 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:11.388923 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.388931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.389604 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.389607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:11.389975 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.389981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.390068 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.390612 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:11.390625 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:11.392217 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
W20251025 20:09:11.392426 28632 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:11.388002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:33883 (local address 127.18.82.126:43255)
1025 20:09:11.388114 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:11.388119 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:11.388157 (+ 38us) server_negotiation.cc:408] Connection header received
1025 20:09:11.388316 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:11.388319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:11.388372 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:11.388448 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:11.389031 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.389471 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.390074 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.390211 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.392334 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:33883: BlockingRecv error: recv got EOF from 127.18.82.66:33883 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:11 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:11.550895 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:11.546600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:39843)
1025 20:09:11.546854 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:11.546870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:11.546988 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:11.547243 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:11.547246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:11.547442 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:11.547624 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.547629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.548255 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.548258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:11.548584 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:11.548590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.548685 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.549157 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:11.549168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:11.550795 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
W20251025 20:09:11.551051 28632 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:11.546634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:39843 (local address 127.18.82.126:43255)
1025 20:09:11.546770 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:11.546774 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:11.546951 (+ 177us) server_negotiation.cc:408] Connection header received
1025 20:09:11.547093 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:11.547096 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:11.547143 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:11.547208 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:11.547732 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.548156 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:11.548686 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:11.548865 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:11.550920 (+ 2055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:39843: BlockingRecv error: recv got EOF from 127.18.82.67:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:12.238852 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:12.234197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:45287)
1025 20:09:12.234697 (+ 500us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:12.234713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:12.234822 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:12.235059 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:12.235062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:12.235232 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:12.235432 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.235438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.236220 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.236223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:12.236627 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.236636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.236727 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.237186 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:12.237196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:12.238747 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":60,"threads_started":1}
W20251025 20:09:12.239023 28635 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:12.234272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:45287 (local address 127.18.82.126:43255)
1025 20:09:12.234560 (+ 288us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:12.234563 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:12.234766 (+ 203us) server_negotiation.cc:408] Connection header received
1025 20:09:12.234914 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:12.234917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:12.234963 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:12.235037 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:12.235540 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.236125 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.236716 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.236835 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.238884 (+ 2049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:45287: BlockingRecv error: recv got EOF from 127.18.82.65:45287 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:12.397461 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:12.393055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:39107)
1025 20:09:12.393340 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:12.393351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:12.393471 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:12.393738 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:12.393740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:12.393912 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:12.394082 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.394087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.394771 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.394778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:12.395174 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.395181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.395284 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.395813 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:12.395826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:12.397365 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
W20251025 20:09:12.397608 28635 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:12.393128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39107 (local address 127.18.82.126:43255)
1025 20:09:12.393265 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:12.393269 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:12.393434 (+ 165us) server_negotiation.cc:408] Connection header received
1025 20:09:12.393600 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:12.393603 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:12.393645 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:12.393704 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:12.394212 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.394668 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.395281 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.395500 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.397470 (+ 1970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:39107: BlockingRecv error: recv got EOF from 127.18.82.66:39107 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:12 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:12.556097 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:12.551674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:48335)
1025 20:09:12.551967 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:12.551979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:12.552071 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:12.552275 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:12.552278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:12.552440 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:12.552626 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.552632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.553223 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.553225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:12.553610 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:12.553617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.553698 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.554184 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:12.554197 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:12.555961 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
W20251025 20:09:12.556196 28635 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:12.551748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:48335 (local address 127.18.82.126:43255)
1025 20:09:12.551870 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:12.551873 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:12.552046 (+ 173us) server_negotiation.cc:408] Connection header received
1025 20:09:12.552140 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:12.552143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:12.552186 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:12.552246 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:12.552715 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.553152 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:12.553698 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:12.553813 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:12.556105 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:48335: BlockingRecv error: recv got EOF from 127.18.82.67:48335 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:13.243798 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:13.239472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:51291)
1025 20:09:13.239744 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:13.239757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:13.239836 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:13.240050 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:13.240052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:13.240228 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:13.240426 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.240432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.241071 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.241073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:13.241411 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.241417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.241490 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.242044 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:13.242057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:13.243690 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20251025 20:09:13.243990 28640 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:13.239624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:51291 (local address 127.18.82.126:43255)
1025 20:09:13.239828 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:13.239832 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:13.239843 (+ 11us) server_negotiation.cc:408] Connection header received
1025 20:09:13.239899 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:13.239901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:13.239942 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:13.240012 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:13.240529 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.240993 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.241493 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.241712 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.243843 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:51291: BlockingRecv error: recv got EOF from 127.18.82.65:51291 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:13.402859 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:13.398320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:49107)
1025 20:09:13.398569 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:13.398597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:13.398685 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:13.398898 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:13.398901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:13.399055 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:13.399228 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.399233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.400079 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.400082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:13.400425 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.400430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.400506 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.400975 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:13.400986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:13.402758 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":62,"threads_started":1}
W20251025 20:09:13.402983 28640 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:13.398400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:49107 (local address 127.18.82.126:43255)
1025 20:09:13.398524 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:13.398527 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:13.398656 (+ 129us) server_negotiation.cc:408] Connection header received
1025 20:09:13.398756 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:13.398758 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:13.398801 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:13.398865 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:13.399317 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.399994 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.400506 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.400631 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.402863 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:49107: BlockingRecv error: recv got EOF from 127.18.82.66:49107 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:13 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:13.561326 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:13.556868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:52427)
1025 20:09:13.557111 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:13.557122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:13.557212 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:13.557477 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:13.557479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:13.557669 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:13.557846 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.557853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.558472 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.558474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:13.558822 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:13.558827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.558899 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.559439 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:13.559449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:13.561219 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
W20251025 20:09:13.561444 28640 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:13.556915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:52427 (local address 127.18.82.126:43255)
1025 20:09:13.557044 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:13.557048 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:13.557179 (+ 131us) server_negotiation.cc:408] Connection header received
1025 20:09:13.557329 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:13.557332 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:13.557378 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:13.557449 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:13.557932 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.558390 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:13.558890 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:13.558997 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:13.561315 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:52427: BlockingRecv error: recv got EOF from 127.18.82.67:52427 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:14.253682 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:14.244602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:44665)
1025 20:09:14.244870 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:14.244882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:14.244971 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:14.245207 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:14.245210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:14.245391 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:14.245608 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.245615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.246301 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.246303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:14.246660 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.246666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.246743 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.251896 (+ 5153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:14.251910 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:14.253575 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20251025 20:09:14.253822 28644 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:14.244696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:44665 (local address 127.18.82.126:43255)
1025 20:09:14.244927 (+ 231us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:14.244931 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:14.244943 (+ 12us) server_negotiation.cc:408] Connection header received
1025 20:09:14.245050 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:14.245053 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:14.245114 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:14.245188 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:14.245699 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.246195 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.246745 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.246869 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.253707 (+ 6838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:44665: BlockingRecv error: recv got EOF from 127.18.82.65:44665 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:14.408288 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:14.403743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:39359)
1025 20:09:14.403988 (+ 245us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:14.404000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:14.404094 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:14.404291 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:14.404294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:14.404472 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:14.404669 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.404675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.405333 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.405336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:14.405802 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.405811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.405918 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.406491 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:14.406512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:14.408148 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
W20251025 20:09:14.408346 28644 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:14.403810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39359 (local address 127.18.82.126:43255)
1025 20:09:14.403937 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:14.403940 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:14.404058 (+ 118us) server_negotiation.cc:408] Connection header received
1025 20:09:14.404160 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:14.404162 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:14.404205 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:14.404269 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:14.404754 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.405247 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.405907 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.406124 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.408258 (+ 2134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:39359: BlockingRecv error: recv got EOF from 127.18.82.66:39359 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:14 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:14.566617 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:14.562025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:47911)
1025 20:09:14.562295 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:14.562307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:14.562401 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:14.562615 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:14.562617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:14.562771 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:14.562948 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.562953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.563597 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.563601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:14.564029 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:14.564041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.564147 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.564827 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:14.564839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:14.566501 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
W20251025 20:09:14.566753 28644 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:14.562093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:47911 (local address 127.18.82.126:43255)
1025 20:09:14.562231 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:14.562238 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:14.562367 (+ 129us) server_negotiation.cc:408] Connection header received
1025 20:09:14.562471 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:14.562474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:14.562517 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:14.562578 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:14.563025 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.563500 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:14.564139 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:14.564349 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:14.566626 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:47911: BlockingRecv error: recv got EOF from 127.18.82.67:47911 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:15.259174 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:15.254528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:46971)
1025 20:09:15.254804 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:15.254820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:15.254918 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:15.255185 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:15.255187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:15.255374 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:15.255569 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.255575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.256242 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.256244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:15.256632 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.256638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.256715 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.257239 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:15.257253 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:15.259018 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
W20251025 20:09:15.259245 28648 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:15.254607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:46971 (local address 127.18.82.126:43255)
1025 20:09:15.254855 (+ 248us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:15.254858 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:15.254871 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:09:15.255014 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:15.255016 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:15.255059 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:15.255133 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:15.255668 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.256164 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.256717 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.256839 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.259135 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:46971: BlockingRecv error: recv got EOF from 127.18.82.65:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":57,"threads_started":1}
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:15.414139 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:15.409163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:35267)
1025 20:09:15.409569 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:15.409590 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:15.409677 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:15.409927 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:15.409930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:15.410088 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:15.410256 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.410261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.411307 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.411310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:15.411651 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.411657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.411731 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.412210 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:15.412222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:15.414038 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":100,"threads_started":1}
W20251025 20:09:15.414376 28648 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:15.409273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:35267 (local address 127.18.82.126:43255)
1025 20:09:15.409395 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:15.409399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:15.409662 (+ 263us) server_negotiation.cc:408] Connection header received
1025 20:09:15.409748 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:15.409753 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:15.409820 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:15.409906 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:15.410359 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.411197 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.411754 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.411985 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.414216 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:35267: BlockingRecv error: recv got EOF from 127.18.82.66:35267 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:15 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:15.571933 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:15.567284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:33075)
1025 20:09:15.567616 (+ 332us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:15.567631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:15.567739 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:15.567977 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:15.567980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:15.568185 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:15.568387 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.568393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.569072 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.569075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:15.569470 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:15.569477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.569601 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.570176 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:15.570188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:15.571781 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":145,"threads_started":1}
W20251025 20:09:15.572005 28648 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:15.567472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:33075 (local address 127.18.82.126:43255)
1025 20:09:15.567616 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:15.567622 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:15.567695 (+ 73us) server_negotiation.cc:408] Connection header received
1025 20:09:15.567817 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:15.567822 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:15.567868 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:15.567935 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:15.568496 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.568968 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:15.569577 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:15.569787 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:15.571902 (+ 2115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:33075: BlockingRecv error: recv got EOF from 127.18.82.67:33075 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:16.264595 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:16.260065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:37401)
1025 20:09:16.260333 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:16.260346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:16.260428 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:16.260725 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:16.260728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:16.260920 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:16.261095 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.261100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.261861 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.261864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:16.262205 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.262211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.262308 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.262864 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:16.262877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:16.264483 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20251025 20:09:16.264734 28652 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:16.260131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:37401 (local address 127.18.82.126:43255)
1025 20:09:16.260369 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:16.260372 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:16.260400 (+ 28us) server_negotiation.cc:408] Connection header received
1025 20:09:16.260554 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:16.260557 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:16.260610 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:16.260686 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:16.261212 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.261759 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.262301 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.262512 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.264602 (+ 2090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:37401: BlockingRecv error: recv got EOF from 127.18.82.65:37401 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:16.419328 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:16.414878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:53333)
1025 20:09:16.415167 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:16.415178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:16.415298 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:16.415570 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:16.415573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:16.415766 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:16.415956 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.415962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.416678 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.416682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:16.416998 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.417003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.417097 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.417701 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:16.417714 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:16.419224 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
W20251025 20:09:16.419437 28652 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:16.414927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:53333 (local address 127.18.82.126:43255)
1025 20:09:16.415077 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:16.415082 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:16.415243 (+ 161us) server_negotiation.cc:408] Connection header received
1025 20:09:16.415409 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:16.415413 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:16.415458 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:16.415527 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:16.416082 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.416585 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.417100 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.417280 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.419344 (+ 2064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:53333: BlockingRecv error: recv got EOF from 127.18.82.66:53333 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:16 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:16.576951 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:16.572694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:45735)
1025 20:09:16.572956 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:16.572968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:16.573059 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:16.573256 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:16.573258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:16.573417 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:16.573619 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.573625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.574248 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.574250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:16.574591 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:16.574597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.574674 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.575162 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:16.575174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:16.576835 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20251025 20:09:16.577054 28652 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:16.572754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:45735 (local address 127.18.82.126:43255)
1025 20:09:16.572921 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:16.572926 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:16.573025 (+ 99us) server_negotiation.cc:408] Connection header received
1025 20:09:16.573126 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:16.573129 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:16.573171 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:16.573239 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:16.573722 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.574164 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:16.574691 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:16.574815 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:16.576962 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:45735: BlockingRecv error: recv got EOF from 127.18.82.67:45735 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:17.269541 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:17.265214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:50365)
1025 20:09:17.265482 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:17.265495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:17.265615 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:17.265838 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:17.265840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:17.266008 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:17.266197 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.266203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.266896 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.266899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:17.267276 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.267282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.267365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.267897 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:17.267911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:17.269408 (+ 1497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20251025 20:09:17.269697 28656 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:17.265271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:50365 (local address 127.18.82.126:43255)
1025 20:09:17.265528 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:17.265531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:17.265552 (+ 21us) server_negotiation.cc:408] Connection header received
1025 20:09:17.265695 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:17.265698 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:17.265743 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:17.265817 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:17.266294 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.266806 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.267367 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.267503 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.269566 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:50365: BlockingRecv error: recv got EOF from 127.18.82.65:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:17.424207 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:17.420039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:41381)
1025 20:09:17.420282 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:17.420294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:17.420393 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:17.420587 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:17.420589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:17.420785 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:17.420957 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.420963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.421574 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.421577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:17.421993 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.422002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.422087 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.422586 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:17.422598 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:17.424106 (+ 1508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
W20251025 20:09:17.424321 28656 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:17.420106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:41381 (local address 127.18.82.126:43255)
1025 20:09:17.420242 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:17.420247 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:17.420355 (+ 108us) server_negotiation.cc:408] Connection header received
1025 20:09:17.420457 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:17.420459 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:17.420502 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:17.420564 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:17.421030 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.421454 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.422092 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.422222 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.424218 (+ 1996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:41381: BlockingRecv error: recv got EOF from 127.18.82.66:41381 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:17 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:17.582465 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:17.577681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:44829)
1025 20:09:17.577981 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:17.577993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:17.578105 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:17.578362 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:17.578365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:17.578586 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:17.578799 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.578806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.579509 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.579512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:17.579901 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:17.579908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.580011 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.580523 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:17.580536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:17.582359 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
W20251025 20:09:17.582585 28656 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:17.577727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:44829 (local address 127.18.82.126:43255)
1025 20:09:17.577889 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:17.577893 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:17.578064 (+ 171us) server_negotiation.cc:408] Connection header received
1025 20:09:17.578190 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:17.578195 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:17.578241 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:17.578310 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:17.578910 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.579401 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:17.580007 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:17.580200 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:17.582479 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:44829: BlockingRecv error: recv got EOF from 127.18.82.67:44829 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:18.274742 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:18.270412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:40127)
1025 20:09:18.270678 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:18.270690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:18.270788 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:18.271005 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:18.271007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:18.271186 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:18.271418 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.271423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.272058 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.272060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:18.272398 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.272404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.272483 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.272998 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:18.273011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:18.274607 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
W20251025 20:09:18.274833 28660 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:18.270475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:40127 (local address 127.18.82.126:43255)
1025 20:09:18.270688 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:18.270691 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:18.270754 (+ 63us) server_negotiation.cc:408] Connection header received
1025 20:09:18.270858 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:18.270861 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:18.270901 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:18.270971 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:18.271510 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.271973 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.272481 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.272603 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.274727 (+ 2124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:40127: BlockingRecv error: recv got EOF from 127.18.82.65:40127 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:18.429309 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:18.424894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:38915)
1025 20:09:18.425170 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:18.425182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:18.425273 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:18.425484 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:18.425486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:18.425691 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:18.425881 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.425886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.426549 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.426551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:18.426924 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.426930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.427014 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.427556 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:18.427574 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:18.429155 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
W20251025 20:09:18.429366 28660 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:18.425009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:38915 (local address 127.18.82.126:43255)
1025 20:09:18.425162 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:18.425166 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:18.425238 (+ 72us) server_negotiation.cc:408] Connection header received
1025 20:09:18.425340 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:18.425342 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:18.425383 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:18.425446 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:18.425969 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.426461 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.427013 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.427143 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.429270 (+ 2127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:38915: BlockingRecv error: recv got EOF from 127.18.82.66:38915 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:18 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:18.587466 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:18.583097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:46351)
1025 20:09:18.583362 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:18.583380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:18.583473 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:18.583708 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:18.583710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:18.583873 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:18.584056 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.584063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.584678 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.584681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:18.585030 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:18.585035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.585116 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.585691 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:18.585704 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:18.587340 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
W20251025 20:09:18.587559 28660 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:18.583166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:46351 (local address 127.18.82.126:43255)
1025 20:09:18.583294 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:18.583297 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:18.583440 (+ 143us) server_negotiation.cc:408] Connection header received
1025 20:09:18.583557 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:18.583560 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:18.583608 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:18.583673 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:18.584140 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.584604 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:18.585103 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:18.585214 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:18.587470 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:46351: BlockingRecv error: recv got EOF from 127.18.82.67:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:19.283547 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:19.275752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:44331)
1025 20:09:19.276036 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:19.276049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:19.276139 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:19.276374 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:19.276376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:19.276546 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:19.276764 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.276770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.277423 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.277425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:19.277849 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.277858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.277941 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.281736 (+ 3795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:19.281750 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:19.283399 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20251025 20:09:19.283638 28664 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:19.275825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:44331 (local address 127.18.82.126:43255)
1025 20:09:19.276074 (+ 249us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:19.276078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:19.276091 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:09:19.276213 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:19.276216 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:19.276260 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:19.276339 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:19.276866 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.277325 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.277948 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.278075 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.283517 (+ 5442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:44331: BlockingRecv error: recv got EOF from 127.18.82.65:44331 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":63,"threads_started":1}
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:19.434254 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:19.430029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:59635)
1025 20:09:19.430266 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:19.430277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:19.430376 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:19.430603 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:19.430605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:19.430779 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:19.430955 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.430961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.431554 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.431556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:19.431898 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.431903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.431978 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.432470 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:19.432481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:19.434129 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
W20251025 20:09:19.434382 28664 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:19.430093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:59635 (local address 127.18.82.126:43255)
1025 20:09:19.430216 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:19.430219 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:19.430336 (+ 117us) server_negotiation.cc:408] Connection header received
1025 20:09:19.430444 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:19.430446 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:19.430487 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:19.430572 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:19.431035 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.431474 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.431977 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.432096 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.434256 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:59635: BlockingRecv error: recv got EOF from 127.18.82.66:59635 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:19 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.67@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:19.592432 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:19.588110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.67:41803)
1025 20:09:19.588384 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:19.588399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:19.588523 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:19.588763 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:19.588766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:19.588959 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:19.589150 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.589156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.589857 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.589863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:19.590203 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:19.590209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.590312 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.590846 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:19.590858 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:19.592335 (+ 1477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20251025 20:09:19.592550 28664 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:19.588200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:41803 (local address 127.18.82.126:43255)
1025 20:09:19.588345 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:19.588350 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:19.588480 (+ 130us) server_negotiation.cc:408] Connection header received
1025 20:09:19.588620 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:19.588623 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:19.588667 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:19.588731 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:19.589264 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.589761 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:19.590312 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:19.590528 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:19.592437 (+ 1909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.67:41803: BlockingRecv error: recv got EOF from 127.18.82.67:41803 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 25 20:09:20 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:20 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.65@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:20.288836 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:20.284274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.65:39843)
1025 20:09:20.284520 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:20.284533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:20.284631 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:20.284825 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:20.284827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:20.284973 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:20.285141 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.285146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.285994 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.286001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:20.286482 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.286496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.286619 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.287169 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:20.287183 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:20.288705 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20251025 20:09:20.288991 28668 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:20.284319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:39843 (local address 127.18.82.126:43255)
1025 20:09:20.284544 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:20.284547 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:20.284586 (+ 39us) server_negotiation.cc:408] Connection header received
1025 20:09:20.284697 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:20.284699 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:20.284740 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:20.284806 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:20.285245 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.285884 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.286602 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.286742 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.288845 (+ 2103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.65:39843: BlockingRecv error: recv got EOF from 127.18.82.65:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
Oct 25 20:09:20 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
Oct 25 20:09:20 dist-test-slave-d2pk krb5kdc[27600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.82.66@KRBTEST.COM for oryx/127.18.82.126@KRBTEST.COM, Server not found in Kerberos database
I20251025 20:09:20.439260 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:20.434988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:43255 (local address 127.18.82.66:56459)
1025 20:09:20.435224 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:20.435235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:20.435328 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:20.435511 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:20.435513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:20.435745 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:20.435938 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.435945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.436662 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.436665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:20.437081 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.437089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.437172 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.437696 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:20.437709 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:20.439160 (+ 1451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.82.126:43255: Server oryx/127.18.82.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
W20251025 20:09:20.439359 28668 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:20.435048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:56459 (local address 127.18.82.126:43255)
1025 20:09:20.435158 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:20.435161 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:20.435287 (+ 126us) server_negotiation.cc:408] Connection header received
1025 20:09:20.435391 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:20.435394 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:20.435435 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:20.435490 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:20.436042 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.436572 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.437182 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.437312 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.439270 (+ 1958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.82.66:56459: BlockingRecv error: recv got EOF from 127.18.82.66:56459 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 20:09:20.510206 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28192
I20251025 20:09:20.516614 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28344
I20251025 20:09:20.521620 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28477
I20251025 20:09:20.527011 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28122
2025-10-25T20:09:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18665 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-25T20:09:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:20Z Disabled control of system clock
I20251025 20:09:20.554876 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44405
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251025 20:09:20.650350 28673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:20.650511 28673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:20.650547 28673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:20.653260 28673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:20.653303 28673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:20.653317 28673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:20.653340 28673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:20.653362 28673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:20.656651 28673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:44405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:20.657490 28673 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:20.658460 28673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:20.663686 28681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:20.663826 28678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:20.663780 28673 server_base.cc:1047] running on GCE node
W20251025 20:09:20.663926 28679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:20.664335 28673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:20.665069 28673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:20.666211 28673 hybrid_clock.cc:648] HybridClock initialized: now 1761422960666199 us; error 26 us; skew 500 ppm
I20251025 20:09:20.667760 28673 webserver.cc:492] Webserver started at http://127.18.82.126:46451/ using document root <none> and password file <none>
I20251025 20:09:20.668223 28673 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:20.668279 28673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:20.668440 28673 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:20.669895 28673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "4114cde94d1c479abae44e80b041ffb6"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "2e279cd85ab0d211b67460fa9890fcd2"
server_key_iv: "3db225cbe4157e719b82609a1055153d"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.670326 28673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "4114cde94d1c479abae44e80b041ffb6"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "2e279cd85ab0d211b67460fa9890fcd2"
server_key_iv: "3db225cbe4157e719b82609a1055153d"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.673327 28673 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:09:20.675272 28687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:20.676167 28673 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:09:20.676272 28673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "4114cde94d1c479abae44e80b041ffb6"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "2e279cd85ab0d211b67460fa9890fcd2"
server_key_iv: "3db225cbe4157e719b82609a1055153d"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.676357 28673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:20.706768 28673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:20.707332 28673 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:20.707499 28673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:20.713762 28739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:44405 every 8 connection(s)
I20251025 20:09:20.713761 28673 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:44405
I20251025 20:09:20.714653 28673 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:09:20.717258 28740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:20.719416 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28673
I20251025 20:09:20.719564 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:09:20.719902 18761 external_mini_cluster.cc:1468] Setting key 040db6f2709af83b9c5e4ad0b2bad6f8
I20251025 20:09:20.722707 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6: Bootstrap starting.
I20251025 20:09:20.724541 28740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:20.725315 28740 log.cc:826] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:20.726634 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:20.720981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.18.82.126:44405)
1025 20:09:20.721533 (+ 552us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:20.721547 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:20.721583 (+ 36us) server_negotiation.cc:408] Connection header received
1025 20:09:20.722053 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:20.722068 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:20.722363 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:20.722632 (+ 269us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:20.723048 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.723696 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.724277 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.724517 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.724984 (+ 467us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:20.725001 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:20.725012 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:20.725423 (+ 411us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:20.725468 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:20.725489 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:20.725625 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:20.725758 (+ 133us) server_negotiation.cc:300] Negotiation successful
1025 20:09:20.725974 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20251025 20:09:20.727267 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6: No bootstrap required, opened a new log
I20251025 20:09:20.729956 28740 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } }
I20251025 20:09:20.730119 28740 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:20.730157 28740 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4114cde94d1c479abae44e80b041ffb6, State: Initialized, Role: FOLLOWER
I20251025 20:09:20.730485 28740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } }
I20251025 20:09:20.730630 28740 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:09:20.730671 28740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:09:20.730731 28740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:20.731356 28740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } }
I20251025 20:09:20.731622 28740 leader_election.cc:304] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4114cde94d1c479abae44e80b041ffb6; no voters:
I20251025 20:09:20.731868 28740 leader_election.cc:290] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:09:20.731992 28745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:09:20.732182 28745 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [term 1 LEADER]: Becoming Leader. State: Replica: 4114cde94d1c479abae44e80b041ffb6, State: Running, Role: LEADER
I20251025 20:09:20.732487 28745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } }
I20251025 20:09:20.732883 28740 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:09:20.733860 28747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4114cde94d1c479abae44e80b041ffb6. Latest consensus state: current_term: 1 leader_uuid: "4114cde94d1c479abae44e80b041ffb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } } }
I20251025 20:09:20.733978 28747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:20.734349 28754 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:09:20.734401 28746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4114cde94d1c479abae44e80b041ffb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4114cde94d1c479abae44e80b041ffb6" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44405 } } }
I20251025 20:09:20.734921 28746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:20.737128 28754 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:09:20.741286 28754 catalog_manager.cc:1357] Generated new cluster ID: cee5ed1cffad49df920cd62e280685a2
I20251025 20:09:20.741345 28754 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:09:20.756053 28754 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:09:20.756953 28754 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:09:20.765427 28754 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4114cde94d1c479abae44e80b041ffb6: Generated new TSK 0
I20251025 20:09:20.766013 28754 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:09:20.768446 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--builtin_ntp_servers=127.18.82.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251025 20:09:20.884333 28764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:20.884526 28764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:20.884595 28764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:20.887703 28764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:20.887746 28764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:20.887830 28764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:09:20.891149 28764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:20.891996 28764 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:20.892949 28764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:20.898566 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:20.898684 28764 server_base.cc:1047] running on GCE node
W20251025 20:09:20.898790 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:20.898572 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:20.899192 28764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:20.899688 28764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:20.900839 28764 hybrid_clock.cc:648] HybridClock initialized: now 1761422960900825 us; error 25 us; skew 500 ppm
I20251025 20:09:20.902805 28764 webserver.cc:492] Webserver started at http://127.18.82.65:46523/ using document root <none> and password file <none>
I20251025 20:09:20.903421 28764 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:20.903493 28764 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:20.903694 28764 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:20.905619 28764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "07a9d662855c4049be83a716c6595c81"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "eb78b225cca0276ae9b6ec6b56993c6c"
server_key_iv: "f4790715b089c2a5d5c370cb33da81b5"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.906160 28764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "07a9d662855c4049be83a716c6595c81"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "eb78b225cca0276ae9b6ec6b56993c6c"
server_key_iv: "f4790715b089c2a5d5c370cb33da81b5"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.909904 28764 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:09:20.912231 28778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:20.913339 28764 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 20:09:20.913444 28764 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "07a9d662855c4049be83a716c6595c81"
format_stamp: "Formatted at 2025-10-25 20:09:20 on dist-test-slave-d2pk"
server_key: "eb78b225cca0276ae9b6ec6b56993c6c"
server_key_iv: "f4790715b089c2a5d5c370cb33da81b5"
server_key_version: "encryptionkey@0"
I20251025 20:09:20.913563 28764 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:20.927546 28764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:20.927986 28764 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:20.928098 28764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:20.928496 28764 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:20.929265 28764 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:20.929306 28764 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:20.929337 28764 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:20.929355 28764 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:20.937644 28764 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:37319
I20251025 20:09:20.937708 28891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:37319 every 8 connection(s)
I20251025 20:09:20.938449 28764 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:09:20.944021 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28764
I20251025 20:09:20.944123 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:09:20.944356 18761 external_mini_cluster.cc:1468] Setting key c152980fe68a0d40c39cc6417cb31646
I20251025 20:09:20.945259 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:20.939873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:53099 (local address 127.18.82.126:44405)
1025 20:09:20.940020 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:20.940024 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:20.940831 (+ 807us) server_negotiation.cc:408] Connection header received
1025 20:09:20.941689 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:20.941693 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:20.941746 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:20.941826 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:20.942670 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.943093 (+ 423us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.943576 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.943674 (+ 98us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.944808 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:20.944823 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:20.944825 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:20.944837 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:20.944853 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:20.944859 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:20.944911 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:20.945078 (+ 167us) server_negotiation.cc:300] Negotiation successful
1025 20:09:20.945132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 20:09:20.945973 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:20.940125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44405 (local address 127.18.82.65:53099)
1025 20:09:20.940689 (+ 564us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:20.940715 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:20.941429 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:20.941965 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:20.941972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:20.942041 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:20.942559 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.942569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.943182 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:20.943184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:20.943507 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:20.943513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:20.943648 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:20.944612 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:20.944613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:20.944639 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:20.944684 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:20.944946 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:20.944953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:20.944958 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 20:09:20.945067 (+ 109us) client_negotiation.cc:240] Negotiation successful
1025 20:09:20.945213 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20251025 20:09:20.946403 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--builtin_ntp_servers=127.18.82.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251025 20:09:20.947029 28892 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44405
I20251025 20:09:20.947208 28892 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:20.947635 28892 heartbeater.cc:507] Master 127.18.82.126:44405 requested a full tablet report, sending...
I20251025 20:09:20.948849 28704 ts_manager.cc:194] Registered new tserver with Master: 07a9d662855c4049be83a716c6595c81 (127.18.82.65:37319)
I20251025 20:09:20.949856 28704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.82.65:53099
W20251025 20:09:21.039628 28895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.039799 28895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.039862 28895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.042735 28895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.042778 28895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.042865 28895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:09:21.046061 28895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.047127 28895 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.048055 28895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.053359 28903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.053373 28900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.053483 28901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.053599 28895 server_base.cc:1047] running on GCE node
I20251025 20:09:21.053894 28895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.054325 28895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.055461 28895 hybrid_clock.cc:648] HybridClock initialized: now 1761422961055450 us; error 26 us; skew 500 ppm
I20251025 20:09:21.056938 28895 webserver.cc:492] Webserver started at http://127.18.82.66:34289/ using document root <none> and password file <none>
I20251025 20:09:21.057390 28895 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.057436 28895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.057621 28895 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.058925 28895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "c2905bacc4e444c08335dda6dace2537"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "0b79bd6fca8eb3adf27679435ef9fa00"
server_key_iv: "a745b7c11c9f2c3bd9815737117366ad"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.059307 28895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "c2905bacc4e444c08335dda6dace2537"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "0b79bd6fca8eb3adf27679435ef9fa00"
server_key_iv: "a745b7c11c9f2c3bd9815737117366ad"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.062139 28895 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:09:21.064183 28909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.064955 28895 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:09:21.065044 28895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "c2905bacc4e444c08335dda6dace2537"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "0b79bd6fca8eb3adf27679435ef9fa00"
server_key_iv: "a745b7c11c9f2c3bd9815737117366ad"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.065115 28895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.082620 28895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.083117 28895 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.083258 28895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.083729 28895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:21.084579 28895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:21.084628 28895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.084681 28895 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:21.084717 28895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.092545 28895 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:38987
I20251025 20:09:21.092576 29022 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:38987 every 8 connection(s)
I20251025 20:09:21.093290 28895 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:09:21.099742 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.094695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39717 (local address 127.18.82.126:44405)
1025 20:09:21.094857 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.094862 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.095548 (+ 686us) server_negotiation.cc:408] Connection header received
1025 20:09:21.096263 (+ 715us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.096266 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.096314 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.096385 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.097219 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.097698 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:21.098230 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.098349 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:21.099379 (+ 1030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.099396 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.099398 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.099410 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.099428 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.099434 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.099490 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.099599 (+ 109us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.099643 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 20:09:21.100239 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.094924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44405 (local address 127.18.82.66:39717)
1025 20:09:21.095438 (+ 514us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:21.095464 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:21.096110 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:21.096525 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:21.096532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:21.096609 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:21.097118 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:21.097129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.097798 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:21.097802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:21.098142 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:21.098149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.098297 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:21.099200 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:21.099202 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:21.099227 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:21.099269 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:21.099496 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:21.099500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:21.099503 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:09:21.099597 (+ 94us) client_negotiation.cc:240] Negotiation successful
1025 20:09:21.099740 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20251025 20:09:21.101111 29023 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44405
I20251025 20:09:21.101239 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 28895
I20251025 20:09:21.101308 29023 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:21.101338 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:09:21.101593 18761 external_mini_cluster.cc:1468] Setting key 21539745e0a49987d85c536974d3d02a
I20251025 20:09:21.101812 29023 heartbeater.cc:507] Master 127.18.82.126:44405 requested a full tablet report, sending...
I20251025 20:09:21.102686 28703 ts_manager.cc:194] Registered new tserver with Master: c2905bacc4e444c08335dda6dace2537 (127.18.82.66:38987)
I20251025 20:09:21.103318 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--builtin_ntp_servers=127.18.82.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251025 20:09:21.103380 28703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.82.66:39717
W20251025 20:09:21.198357 29026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.198519 29026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.198586 29026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.201260 29026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.201303 29026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.201412 29026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:09:21.204551 29026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.205369 29026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.206367 29026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.211694 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.211721 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.211715 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.211879 29026 server_base.cc:1047] running on GCE node
I20251025 20:09:21.212188 29026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.212630 29026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.213768 29026 hybrid_clock.cc:648] HybridClock initialized: now 1761422961213759 us; error 20 us; skew 500 ppm
I20251025 20:09:21.215144 29026 webserver.cc:492] Webserver started at http://127.18.82.67:40987/ using document root <none> and password file <none>
I20251025 20:09:21.215582 29026 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.215628 29026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.215785 29026 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.217183 29026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "81a52fc622c34b2cb33b8adf659b98b4"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "1fd5125679ab7a08423762f5be35d1df"
server_key_iv: "58ee5efb03880f3366b0b870be767e57"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.217607 29026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "81a52fc622c34b2cb33b8adf659b98b4"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "1fd5125679ab7a08423762f5be35d1df"
server_key_iv: "58ee5efb03880f3366b0b870be767e57"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.220373 29026 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20251025 20:09:21.222183 29040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.223049 29026 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:21.223132 29026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "81a52fc622c34b2cb33b8adf659b98b4"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "1fd5125679ab7a08423762f5be35d1df"
server_key_iv: "58ee5efb03880f3366b0b870be767e57"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.223197 29026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.231892 29026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.232327 29026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.232439 29026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.232824 29026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:21.233654 29026 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:21.233700 29026 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.233732 29026 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:21.233752 29026 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.241499 29026 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34815
I20251025 20:09:21.241549 29153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34815 every 8 connection(s)
I20251025 20:09:21.242308 29026 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:09:21.248016 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29026
I20251025 20:09:21.248113 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:09:21.248351 18761 external_mini_cluster.cc:1468] Setting key 35ff387c53815022681d48df941ffbf5
I20251025 20:09:21.249172 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.243753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:46037 (local address 127.18.82.126:44405)
1025 20:09:21.243880 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.243884 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.244606 (+ 722us) server_negotiation.cc:408] Connection header received
1025 20:09:21.245314 (+ 708us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.245318 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.245365 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.245435 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.246347 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.246824 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:21.247422 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.247543 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:21.248748 (+ 1205us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.248764 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.248766 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.248777 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.248794 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.248800 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.248859 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.248999 (+ 140us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.249050 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:09:21.249725 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.243928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44405 (local address 127.18.82.67:46037)
1025 20:09:21.244481 (+ 553us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:21.244518 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:21.245143 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:21.245633 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:21.245640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:21.245710 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:21.246212 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:21.246222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.246919 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:21.246922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:21.247339 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:21.247345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:21.247524 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:21.248549 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:21.248551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:21.248580 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:21.248622 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:21.248871 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:21.248880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:21.248886 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 20:09:21.248995 (+ 109us) client_negotiation.cc:240] Negotiation successful
1025 20:09:21.249169 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20251025 20:09:21.250604 29154 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44405
I20251025 20:09:21.250775 29154 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:21.251158 29154 heartbeater.cc:507] Master 127.18.82.126:44405 requested a full tablet report, sending...
I20251025 20:09:21.251981 28703 ts_manager.cc:194] Registered new tserver with Master: 81a52fc622c34b2cb33b8adf659b98b4 (127.18.82.67:34815)
I20251025 20:09:21.252468 28703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.82.67:46037
I20251025 20:09:21.261612 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 20:09:21.263775 28743 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:21.263170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.18.82.126:44405)
1025 20:09:21.263264 (+ 94us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.263267 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.263386 (+ 119us) server_negotiation.cc:408] Connection header received
1025 20:09:21.263470 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.263472 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.263504 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.263561 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.263707 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34482: BlockingRecv error: recv got EOF from 127.0.0.1:34482 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20251025 20:09:21.263962 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28764
I20251025 20:09:21.269950 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28895
I20251025 20:09:21.274917 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29026
I20251025 20:09:21.279767 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 28673
2025-10-25T20:09:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (751 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-25T20:09:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:21Z Disabled control of system clock
I20251025 20:09:21.306428 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44437
--webserver_interface=127.18.82.126
--webserver_port=0
--builtin_ntp_servers=127.18.82.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:44437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 20:09:21.402269 29165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.402432 29165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.402478 29165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.405383 29165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:21.405431 29165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.405448 29165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.405488 29165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:21.405535 29165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:21.408951 29165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:44437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.409852 29165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.410789 29165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.415464 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.415555 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.415540 29165 server_base.cc:1047] running on GCE node
W20251025 20:09:21.415759 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.416211 29165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.416996 29165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.418129 29165 hybrid_clock.cc:648] HybridClock initialized: now 1761422961418111 us; error 32 us; skew 500 ppm
I20251025 20:09:21.419529 29165 webserver.cc:492] Webserver started at http://127.18.82.126:37119/ using document root <none> and password file <none>
I20251025 20:09:21.419996 29165 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.420042 29165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.420181 29165 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.421907 29165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "c6b9b11db61d49349bb8c55e631fb604"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "d1e4e3fd015e5b3bc3bd5c6a3caa77bd"
server_key_iv: "7e868ce2727c36ab303d2571745bd37d"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.422292 29165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "c6b9b11db61d49349bb8c55e631fb604"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "d1e4e3fd015e5b3bc3bd5c6a3caa77bd"
server_key_iv: "7e868ce2727c36ab303d2571745bd37d"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.425340 29165 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:09:21.427338 29179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.428165 29165 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:21.428259 29165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "c6b9b11db61d49349bb8c55e631fb604"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "d1e4e3fd015e5b3bc3bd5c6a3caa77bd"
server_key_iv: "7e868ce2727c36ab303d2571745bd37d"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.428325 29165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.432215 29165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.432670 29165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.432783 29165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.439010 29165 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:44437
I20251025 20:09:21.439018 29231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:44437 every 8 connection(s)
I20251025 20:09:21.439890 29165 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:09:21.440236 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29165
I20251025 20:09:21.440361 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:09:21.440646 18761 external_mini_cluster.cc:1468] Setting key fbcec9d72b747111e997764016805d97
I20251025 20:09:21.442713 29232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:21.445107 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.441741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.18.82.126:44437)
1025 20:09:21.442254 (+ 513us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.442262 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.442289 (+ 27us) server_negotiation.cc:408] Connection header received
1025 20:09:21.442881 (+ 592us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.442899 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.443160 (+ 261us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.443457 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.444175 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.444179 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.444188 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.444197 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.444230 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.444240 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.444383 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.444460 (+ 77us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.444636 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
I20251025 20:09:21.447963 29232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604: Bootstrap starting.
I20251025 20:09:21.449764 29232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:21.450358 29232 log.cc:826] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:21.451759 29232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604: No bootstrap required, opened a new log
I20251025 20:09:21.453787 29232 raft_consensus.cc:359] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } }
I20251025 20:09:21.453931 29232 raft_consensus.cc:385] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:21.453992 29232 raft_consensus.cc:740] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9b11db61d49349bb8c55e631fb604, State: Initialized, Role: FOLLOWER
I20251025 20:09:21.454349 29232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } }
I20251025 20:09:21.454456 29232 raft_consensus.cc:399] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:09:21.454511 29232 raft_consensus.cc:493] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:09:21.454594 29232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:21.455390 29232 raft_consensus.cc:515] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } }
I20251025 20:09:21.455679 29232 leader_election.cc:304] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b9b11db61d49349bb8c55e631fb604; no voters:
I20251025 20:09:21.455924 29232 leader_election.cc:290] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:09:21.456060 29237 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:09:21.456240 29237 raft_consensus.cc:697] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [term 1 LEADER]: Becoming Leader. State: Replica: c6b9b11db61d49349bb8c55e631fb604, State: Running, Role: LEADER
I20251025 20:09:21.456525 29237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } }
I20251025 20:09:21.456836 29232 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:09:21.457974 29239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b9b11db61d49349bb8c55e631fb604. Latest consensus state: current_term: 1 leader_uuid: "c6b9b11db61d49349bb8c55e631fb604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } } }
I20251025 20:09:21.457942 29238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b9b11db61d49349bb8c55e631fb604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b9b11db61d49349bb8c55e631fb604" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 44437 } } }
I20251025 20:09:21.458065 29239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:21.458065 29238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:21.458415 29246 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:09:21.461025 29246 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:09:21.465248 29246 catalog_manager.cc:1357] Generated new cluster ID: dc3f9f03e13342e2842d0ba3ce44bef2
I20251025 20:09:21.465349 29246 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:09:21.484537 29246 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:09:21.484777 29246 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:09:21.491261 29246 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c6b9b11db61d49349bb8c55e631fb604: Generated new TSK 0
I20251025 20:09:21.491801 29246 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 20:09:21.504758 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--builtin_ntp_servers=127.18.82.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 20:09:21.602367 29256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.602547 29256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.602618 29256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.605412 29256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.605455 29256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.605572 29256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:09:21.608932 29256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.609848 29256 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.610787 29256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.616364 29262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.616356 29261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.616547 29264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.616724 29256 server_base.cc:1047] running on GCE node
I20251025 20:09:21.617061 29256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.617520 29256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.618685 29256 hybrid_clock.cc:648] HybridClock initialized: now 1761422961618644 us; error 53 us; skew 500 ppm
I20251025 20:09:21.620208 29256 webserver.cc:492] Webserver started at http://127.18.82.65:38863/ using document root <none> and password file <none>
I20251025 20:09:21.620683 29256 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.620733 29256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.620891 29256 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.622309 29256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "9513cccd02a2449a99bf5cb960bfe9c8"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c781e775d9dea1803db6e87c2e26168e"
server_key_iv: "4d7ef2f604ef5e90b1cd16195848f04b"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.622754 29256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "9513cccd02a2449a99bf5cb960bfe9c8"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c781e775d9dea1803db6e87c2e26168e"
server_key_iv: "4d7ef2f604ef5e90b1cd16195848f04b"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.625593 29256 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:09:21.627540 29270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.628363 29256 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:21.628458 29256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "9513cccd02a2449a99bf5cb960bfe9c8"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c781e775d9dea1803db6e87c2e26168e"
server_key_iv: "4d7ef2f604ef5e90b1cd16195848f04b"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.628540 29256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.632561 29256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.633021 29256 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.633153 29256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.633621 29256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:21.634420 29256 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:21.634465 29256 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.634516 29256 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:21.634552 29256 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.642292 29256 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:44491
I20251025 20:09:21.642324 29383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:44491 every 8 connection(s)
I20251025 20:09:21.643112 29256 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:09:21.647564 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:60761 (local address 127.18.82.126:44437)
1025 20:09:21.644674 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.644679 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.645424 (+ 745us) server_negotiation.cc:408] Connection header received
1025 20:09:21.646121 (+ 697us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.646124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.646161 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.646231 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.647184 (+ 953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.647186 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.647189 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.647197 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.647212 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.647218 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.647283 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.647395 (+ 112us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.647428 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 20:09:21.648154 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.644785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44437 (local address 127.18.82.65:60761)
1025 20:09:21.645305 (+ 520us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:21.645335 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:21.645962 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:21.646360 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:21.646367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:21.646439 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:21.646984 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:21.646985 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:21.647015 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:21.647064 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:21.647280 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:21.647284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:21.647287 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:09:21.647428 (+ 141us) client_negotiation.cc:240] Negotiation successful
1025 20:09:21.647636 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
I20251025 20:09:21.649286 29384 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44437
I20251025 20:09:21.649453 29384 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:21.649751 29384 heartbeater.cc:507] Master 127.18.82.126:44437 requested a full tablet report, sending...
I20251025 20:09:21.649768 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29256
I20251025 20:09:21.649878 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:09:21.650077 18761 external_mini_cluster.cc:1468] Setting key edabcd5ff3f48baa179cc256040c3ca4
I20251025 20:09:21.650995 29196 ts_manager.cc:194] Registered new tserver with Master: 9513cccd02a2449a99bf5cb960bfe9c8 (127.18.82.65:44491)
I20251025 20:09:21.651778 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--builtin_ntp_servers=127.18.82.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 20:09:21.746824 29387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.746995 29387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.747040 29387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.749792 29387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.749830 29387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.749905 29387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:09:21.752946 29387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.753780 29387 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.754684 29387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.760408 29392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.760464 29393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.760533 29387 server_base.cc:1047] running on GCE node
W20251025 20:09:21.760408 29395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.760919 29387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.761346 29387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.762482 29387 hybrid_clock.cc:648] HybridClock initialized: now 1761422961762470 us; error 29 us; skew 500 ppm
I20251025 20:09:21.763871 29387 webserver.cc:492] Webserver started at http://127.18.82.66:37213/ using document root <none> and password file <none>
I20251025 20:09:21.764340 29387 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.764396 29387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.764556 29387 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.766064 29387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "7f6dca980a084125bbb04e9b81daf285"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "657d7192623407c8b9d0103f2d402aee"
server_key_iv: "34d2902a987a8dd3b3dd3f28142e0eaa"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.766527 29387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "7f6dca980a084125bbb04e9b81daf285"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "657d7192623407c8b9d0103f2d402aee"
server_key_iv: "34d2902a987a8dd3b3dd3f28142e0eaa"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.769366 29387 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 20:09:21.771121 29401 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.771908 29387 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:09:21.771997 29387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "7f6dca980a084125bbb04e9b81daf285"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "657d7192623407c8b9d0103f2d402aee"
server_key_iv: "34d2902a987a8dd3b3dd3f28142e0eaa"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.772070 29387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.776031 29387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.776525 29387 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.776671 29387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.777139 29387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:21.778007 29387 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:21.778054 29387 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.778105 29387 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:21.778136 29387 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.786154 29387 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:40647
I20251025 20:09:21.786190 29514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:40647 every 8 connection(s)
I20251025 20:09:21.787007 29387 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:09:21.791633 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.788578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:39431 (local address 127.18.82.126:44437)
1025 20:09:21.788709 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.788712 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.789439 (+ 727us) server_negotiation.cc:408] Connection header received
1025 20:09:21.790136 (+ 697us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.790140 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.790187 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.790257 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.791202 (+ 945us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.791206 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.791209 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.791220 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.791240 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.791246 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.791319 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.791458 (+ 139us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.791498 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:09:21.792227 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.788785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44437 (local address 127.18.82.66:39431)
1025 20:09:21.789309 (+ 524us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:21.789345 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:21.789952 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:21.790409 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:21.790416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:21.790485 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:21.790995 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:21.790996 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:21.791023 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:21.791068 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:21.791314 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:21.791329 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:21.791336 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 20:09:21.791508 (+ 172us) client_negotiation.cc:240] Negotiation successful
1025 20:09:21.791684 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20251025 20:09:21.793285 29515 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44437
I20251025 20:09:21.793475 29515 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:21.793819 29515 heartbeater.cc:507] Master 127.18.82.126:44437 requested a full tablet report, sending...
I20251025 20:09:21.794659 29196 ts_manager.cc:194] Registered new tserver with Master: 7f6dca980a084125bbb04e9b81daf285 (127.18.82.66:40647)
I20251025 20:09:21.796341 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29387
I20251025 20:09:21.796452 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:09:21.796692 18761 external_mini_cluster.cc:1468] Setting key 4f575bb8481e2de293fa3a15076a00c4
I20251025 20:09:21.798230 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--builtin_ntp_servers=127.18.82.84:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 20:09:21.892336 29518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:21.892488 29518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:21.892529 29518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:21.895128 29518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:21.895164 29518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:21.895220 29518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:09:21.898383 29518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:21.899194 29518 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:21.900076 29518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:21.905525 29524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.905491 29523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:21.905493 29526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:21.905684 29518 server_base.cc:1047] running on GCE node
I20251025 20:09:21.906030 29518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:21.906517 29518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:21.907670 29518 hybrid_clock.cc:648] HybridClock initialized: now 1761422961907650 us; error 36 us; skew 500 ppm
I20251025 20:09:21.909162 29518 webserver.cc:492] Webserver started at http://127.18.82.67:39475/ using document root <none> and password file <none>
I20251025 20:09:21.909696 29518 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:21.909749 29518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:21.909921 29518 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:21.911413 29518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "ce027eead3f341579c04322155f2a787"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c34eb40a7c55ad440ae20f0cc0e2e758"
server_key_iv: "0985c69b15376b21e55a3b498f213a1e"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.911847 29518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "ce027eead3f341579c04322155f2a787"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c34eb40a7c55ad440ae20f0cc0e2e758"
server_key_iv: "0985c69b15376b21e55a3b498f213a1e"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.914788 29518 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:09:21.916618 29532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.917474 29518 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:21.917593 29518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "ce027eead3f341579c04322155f2a787"
format_stamp: "Formatted at 2025-10-25 20:09:21 on dist-test-slave-d2pk"
server_key: "c34eb40a7c55ad440ae20f0cc0e2e758"
server_key_iv: "0985c69b15376b21e55a3b498f213a1e"
server_key_version: "encryptionkey@0"
I20251025 20:09:21.917667 29518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:21.921609 29518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:21.922063 29518 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:21.922199 29518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:21.922621 29518 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:21.923370 29518 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:21.923415 29518 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.923467 29518 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:21.923494 29518 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:21.931321 29518 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:34369
I20251025 20:09:21.931350 29645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:34369 every 8 connection(s)
I20251025 20:09:21.932070 29518 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:09:21.932883 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29518
I20251025 20:09:21.932993 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:09:21.933202 18761 external_mini_cluster.cc:1468] Setting key e9649e20567f876e20c82526eac8cd72
I20251025 20:09:21.936697 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.933550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:60867 (local address 127.18.82.126:44437)
1025 20:09:21.933742 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.933746 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.934655 (+ 909us) server_negotiation.cc:408] Connection header received
1025 20:09:21.935371 (+ 716us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.935374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.935412 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.935475 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.936392 (+ 917us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:21.936395 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:21.936397 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 20:09:21.936404 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:21.936417 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 20:09:21.936424 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:21.936481 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:21.936571 (+ 90us) server_negotiation.cc:300] Negotiation successful
1025 20:09:21.936601 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251025 20:09:21.937304 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:21.933815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:44437 (local address 127.18.82.67:60867)
1025 20:09:21.934533 (+ 718us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:21.934577 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:21.935210 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:21.935582 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:21.935589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:21.935659 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:21.936213 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 20:09:21.936215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:21.936242 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 20:09:21.936289 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:21.936473 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:21.936477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:21.936480 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 20:09:21.936619 (+ 139us) client_negotiation.cc:240] Negotiation successful
1025 20:09:21.936762 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":111,"threads_started":1}
I20251025 20:09:21.938315 29646 heartbeater.cc:344] Connected to a master server at 127.18.82.126:44437
I20251025 20:09:21.938477 29646 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:21.938802 29646 heartbeater.cc:507] Master 127.18.82.126:44437 requested a full tablet report, sending...
I20251025 20:09:21.939600 29196 ts_manager.cc:194] Registered new tserver with Master: ce027eead3f341579c04322155f2a787 (127.18.82.67:34369)
I20251025 20:09:21.946584 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 20:09:21.948953 29235 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:09:21.948217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.18.82.126:44437)
1025 20:09:21.948339 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:21.948343 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:21.948450 (+ 107us) server_negotiation.cc:408] Connection header received
1025 20:09:21.948573 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:21.948576 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:21.948606 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:21.948670 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:21.948864 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59874: BlockingRecv error: recv got EOF from 127.0.0.1:59874 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:09:21.949090 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29256
I20251025 20:09:21.955194 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29387
I20251025 20:09:21.960742 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29518
I20251025 20:09:21.965363 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29165
2025-10-25T20:09:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (685 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:09:22 dist-test-slave-d2pk krb5kdc[29656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:09:22 dist-test-slave-d2pk krb5kdc[29656](info): set up 2 sockets
Oct 25 20:09:22 dist-test-slave-d2pk krb5kdc[29656](info): commencing operation
krb5kdc: starting...
W20251025 20:09:23.773773 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.788s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:09:23 dist-test-slave-d2pk krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:09:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:23.889201 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35985
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:23.986863 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:23.987044 29672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:23.987099 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:23.989928 29672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:09:23.989972 29672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:23.989992 29672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:23.990008 29672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:09:23.990023 29672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:09:23.993649 29672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:35985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:23.994467 29672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:23.995188 29672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:23.999934 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:23.999971 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:23.999982 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:24.000325 29672 server_base.cc:1047] running on GCE node
I20251025 20:09:24.000766 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:24.001531 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:24.002686 29672 hybrid_clock.cc:648] HybridClock initialized: now 1761422964002668 us; error 35 us; skew 500 ppm
Oct 25 20:09:23 dist-test-slave-d2pk krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422963, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:24.004906 29672 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:09:24.005818 29672 webserver.cc:492] Webserver started at http://127.18.82.126:43757/ using document root <none> and password file <none>
I20251025 20:09:24.006325 29672 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:24.006377 29672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:24.006547 29672 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:24.008025 29672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "361de9900136414892dbfac131963848"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8647618f4b3fac13a0de221a27ee8c89"
server_key_iv: "2c0dee0cd6e477ab18da94859fa7bd18"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.008438 29672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "361de9900136414892dbfac131963848"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8647618f4b3fac13a0de221a27ee8c89"
server_key_iv: "2c0dee0cd6e477ab18da94859fa7bd18"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.011284 29672 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 20:09:24.013062 29687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.013875 29672 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:24.013976 29672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "361de9900136414892dbfac131963848"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8647618f4b3fac13a0de221a27ee8c89"
server_key_iv: "2c0dee0cd6e477ab18da94859fa7bd18"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.014045 29672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:24.030361 29672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:24.032758 29672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:24.032897 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:24.038952 29672 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:35985
I20251025 20:09:24.038967 29739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:35985 every 8 connection(s)
I20251025 20:09:24.039850 29672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:09:24.042217 29740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:24.043349 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29672
I20251025 20:09:24.043470 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:09:24.043728 18761 external_mini_cluster.cc:1468] Setting key ac6d4ba5611586398af408300dc4a6a3
I20251025 20:09:24.048239 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: Bootstrap starting.
I20251025 20:09:24.050400 29740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: Neither blocks nor log segments found. Creating new log.
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:24.051056 29740 log.cc:826] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:24.052685 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: No bootstrap required, opened a new log
I20251025 20:09:24.054796 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.044766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.18.82.126:35985)
1025 20:09:24.045081 (+ 315us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.045089 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.045117 (+ 28us) server_negotiation.cc:408] Connection header received
1025 20:09:24.045666 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.045684 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.045931 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.046217 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:24.047111 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.047941 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.048597 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.048861 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.051004 (+ 2143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:24.051021 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:24.051030 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:24.051057 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:24.052658 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.053034 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.053038 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.053041 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.053101 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.053305 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.053308 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.053310 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.053645 (+ 335us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:24.053810 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.054085 (+ 275us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.054294 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20251025 20:09:24.055596 29740 raft_consensus.cc:359] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } }
I20251025 20:09:24.055806 29740 raft_consensus.cc:385] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:24.055871 29740 raft_consensus.cc:740] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361de9900136414892dbfac131963848, State: Initialized, Role: FOLLOWER
I20251025 20:09:24.056286 29740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } }
I20251025 20:09:24.056437 29740 raft_consensus.cc:399] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:09:24.056501 29740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:09:24.056595 29740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:24.057335 29740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } }
I20251025 20:09:24.057652 29740 leader_election.cc:304] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 361de9900136414892dbfac131963848; no voters:
I20251025 20:09:24.057938 29740 leader_election.cc:290] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:09:24.058224 29745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:09:24.058395 29745 raft_consensus.cc:697] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [term 1 LEADER]: Becoming Leader. State: Replica: 361de9900136414892dbfac131963848, State: Running, Role: LEADER
I20251025 20:09:24.058658 29745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } }
I20251025 20:09:24.058859 29740 sys_catalog.cc:565] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:09:24.059875 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361de9900136414892dbfac131963848. Latest consensus state: current_term: 1 leader_uuid: "361de9900136414892dbfac131963848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } } }
I20251025 20:09:24.059978 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:24.060199 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "361de9900136414892dbfac131963848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361de9900136414892dbfac131963848" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 35985 } } }
I20251025 20:09:24.060338 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848 [sys.catalog]: This master's current role is: LEADER
I20251025 20:09:24.060361 29754 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:09:24.062948 29754 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:09:24.067786 29754 catalog_manager.cc:1357] Generated new cluster ID: 62e8787cc47d49cb85d4abb7c6229a17
I20251025 20:09:24.067837 29754 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:09:24.081002 29754 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:09:24.081986 29754 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:09:24.089794 29754 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: Generated new TSK 0
I20251025 20:09:24.090314 29754 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:24.143980 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35985
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:24.246560 29768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:24.246742 29768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:24.246793 29768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:24.249992 29768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:24.250041 29768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:24.250108 29768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:09:24.253719 29768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:24.254546 29768 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:24.255239 29768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:24.261548 29776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:24.261618 29768 server_base.cc:1047] running on GCE node
W20251025 20:09:24.261627 29774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:24.261564 29773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:24.262056 29768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:24.262528 29768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:24.263674 29768 hybrid_clock.cc:648] HybridClock initialized: now 1761422964263649 us; error 36 us; skew 500 ppm
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:24.265966 29768 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:09:24.266933 29768 webserver.cc:492] Webserver started at http://127.18.82.65:45315/ using document root <none> and password file <none>
I20251025 20:09:24.267453 29768 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:24.267501 29768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:24.267697 29768 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:24.269279 29768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "2fa75d2b46da4c56951a38cdf7f008d0"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8e14b8e280f26ca4bf72429c588d8e35"
server_key_iv: "cdad3ed66695b1e06b2b3891383b5613"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.269721 29768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "2fa75d2b46da4c56951a38cdf7f008d0"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8e14b8e280f26ca4bf72429c588d8e35"
server_key_iv: "cdad3ed66695b1e06b2b3891383b5613"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.272712 29768 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251025 20:09:24.274628 29783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.275460 29768 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:09:24.275553 29768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "2fa75d2b46da4c56951a38cdf7f008d0"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "8e14b8e280f26ca4bf72429c588d8e35"
server_key_iv: "cdad3ed66695b1e06b2b3891383b5613"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.275627 29768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:24.294973 29768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:24.300942 29768 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:24.301142 29768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:24.301652 29768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:24.302399 29768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:24.302446 29768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.302505 29768 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:24.302533 29768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.310308 29768 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:43899
I20251025 20:09:24.310343 29896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:43899 every 8 connection(s)
I20251025 20:09:24.311064 29768 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:24.318939 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29768
I20251025 20:09:24.319068 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:09:24.319320 18761 external_mini_cluster.cc:1468] Setting key a43e92c8aad8468e955868b672a7a41f
I20251025 20:09:24.321350 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.312506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:45289 (local address 127.18.82.126:35985)
1025 20:09:24.312647 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.312651 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.313188 (+ 537us) server_negotiation.cc:408] Connection header received
1025 20:09:24.313854 (+ 666us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.313857 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.313903 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.313965 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:24.314931 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.315362 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.315824 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.315927 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.318303 (+ 2376us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:24.318324 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:24.318327 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:24.318348 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:24.319668 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.320139 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.320145 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.320146 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.320188 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.320466 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.320469 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.320471 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.320601 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:24.320681 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.321105 (+ 424us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.321214 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 20:09:24.321919 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.312736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35985 (local address 127.18.82.65:45289)
1025 20:09:24.313076 (+ 340us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.313105 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.313702 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.314080 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.314086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.314377 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:24.314825 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.314835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.315436 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.315438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.315757 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.315762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.315894 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.316781 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:24.316801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:24.318178 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:24.319781 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.319787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.319797 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.320052 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.320264 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.320269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.320272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.320382 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.320695 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:24.320702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:24.320890 (+ 188us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.321049 (+ 159us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.321268 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20251025 20:09:24.322964 29897 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35985
I20251025 20:09:24.323160 29897 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:24.323601 29897 heartbeater.cc:507] Master 127.18.82.126:35985 requested a full tablet report, sending...
I20251025 20:09:24.325013 29704 ts_manager.cc:194] Registered new tserver with Master: 2fa75d2b46da4c56951a38cdf7f008d0 (127.18.82.65:43899)
I20251025 20:09:24.326653 29704 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:45289
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:24.359808 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35985
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:24.452903 29904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:24.453074 29904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:24.453123 29904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:24.456025 29904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:24.456075 29904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:24.456164 29904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:09:24.459872 29904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:24.460830 29904 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:24.461526 29904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:24.467324 29910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:24.467286 29912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:24.467283 29909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:24.467485 29904 server_base.cc:1047] running on GCE node
I20251025 20:09:24.467785 29904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:24.468199 29904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:24.469338 29904 hybrid_clock.cc:648] HybridClock initialized: now 1761422964469326 us; error 26 us; skew 500 ppm
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:24.471662 29904 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:09:24.472482 29904 webserver.cc:492] Webserver started at http://127.18.82.66:44199/ using document root <none> and password file <none>
I20251025 20:09:24.472977 29904 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:24.473040 29904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:24.473209 29904 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:24.474756 29904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "020e0d109c3e451092094d9abebda679"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "64247ff58a9d36cd92f96e0b1c737441"
server_key_iv: "b2421cdfc7bc5840b913b9fd6b7f6e66"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.475209 29904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "020e0d109c3e451092094d9abebda679"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "64247ff58a9d36cd92f96e0b1c737441"
server_key_iv: "b2421cdfc7bc5840b913b9fd6b7f6e66"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.478221 29904 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 20:09:24.480087 29919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.480935 29904 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 20:09:24.481036 29904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "020e0d109c3e451092094d9abebda679"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "64247ff58a9d36cd92f96e0b1c737441"
server_key_iv: "b2421cdfc7bc5840b913b9fd6b7f6e66"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.481113 29904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:24.506589 29904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:24.508898 29904 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:24.509058 29904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:24.509539 29904 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:24.510326 29904 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:24.510370 29904 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.510408 29904 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:24.510442 29904 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.518690 29904 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:36425
I20251025 20:09:24.518729 30032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:36425 every 8 connection(s)
I20251025 20:09:24.519483 29904 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:09:24.524372 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 29904
I20251025 20:09:24.524499 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:09:24.524750 18761 external_mini_cluster.cc:1468] Setting key 4e0e55dfa0b71ce7b8d3442136595e6b
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:24.530675 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.520976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:45851 (local address 127.18.82.126:35985)
1025 20:09:24.521123 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.521127 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.521707 (+ 580us) server_negotiation.cc:408] Connection header received
1025 20:09:24.522413 (+ 706us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.522417 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.522465 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.522545 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:24.523543 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.524024 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.524611 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.524805 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.527396 (+ 2591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:24.527422 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:24.527424 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:24.527446 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:24.528787 (+ 1341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.529378 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.529382 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.529385 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.529430 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.529743 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.529746 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.529747 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.529871 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:24.529939 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.530488 (+ 549us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.530571 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 20:09:24.531579 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.521182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35985 (local address 127.18.82.66:45851)
1025 20:09:24.521592 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.521624 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.522232 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.522660 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.522667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.522972 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:24.523441 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.523452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.524101 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.524103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.524483 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.524491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.524696 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.525804 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:24.525836 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:24.527240 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:24.528917 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.528926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.528938 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.529266 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.529529 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.529535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.529538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.529657 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.529973 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:24.529982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:24.530245 (+ 263us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.530457 (+ 212us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.530798 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20251025 20:09:24.532496 30033 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35985
I20251025 20:09:24.532699 30033 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:24.533087 30033 heartbeater.cc:507] Master 127.18.82.126:35985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:09:24.533951 29704 ts_manager.cc:194] Registered new tserver with Master: 020e0d109c3e451092094d9abebda679 (127.18.82.66:36425)
I20251025 20:09:24.534459 29704 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:45851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:09:24.564038 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:35985
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:09:24.658613 30040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:09:24.658799 30040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:09:24.658878 30040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:09:24.662039 30040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:09:24.662089 30040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:09:24.662189 30040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:09:24.665833 30040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:09:24.666756 30040 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:09:24.667502 30040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:09:24.673087 30045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:24.673163 30046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:09:24.673158 30048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:09:24.673473 30040 server_base.cc:1047] running on GCE node
I20251025 20:09:24.673811 30040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:09:24.674252 30040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:09:24.675391 30040 hybrid_clock.cc:648] HybridClock initialized: now 1761422964675371 us; error 33 us; skew 500 ppm
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:09:24.677915 30040 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:09:24.678798 30040 webserver.cc:492] Webserver started at http://127.18.82.67:43747/ using document root <none> and password file <none>
I20251025 20:09:24.679302 30040 fs_manager.cc:362] Metadata directory not provided
I20251025 20:09:24.679351 30040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:09:24.679498 30040 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 20:09:24.680914 30040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "804f9b98d4244034a0ac45cee2b002f6"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "d0591a92cbb7b6f6140ee3b054580b66"
server_key_iv: "a5f98c01e744e844f20dd9630d02fc16"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.681326 30040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "804f9b98d4244034a0ac45cee2b002f6"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "d0591a92cbb7b6f6140ee3b054580b66"
server_key_iv: "a5f98c01e744e844f20dd9630d02fc16"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.684221 30040 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:09:24.686127 30055 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.687027 30040 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:09:24.687111 30040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "804f9b98d4244034a0ac45cee2b002f6"
format_stamp: "Formatted at 2025-10-25 20:09:24 on dist-test-slave-d2pk"
server_key: "d0591a92cbb7b6f6140ee3b054580b66"
server_key_iv: "a5f98c01e744e844f20dd9630d02fc16"
server_key_version: "encryptionkey@0"
I20251025 20:09:24.687178 30040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:09:24.710063 30040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:09:24.712252 30040 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:09:24.712373 30040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:09:24.712790 30040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:09:24.713486 30040 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:09:24.713572 30040 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.713613 30040 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:09:24.713631 30040 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:09:24.721621 30040 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:43839
I20251025 20:09:24.721653 30168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:43839 every 8 connection(s)
I20251025 20:09:24.722398 30040 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:09:24.728268 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 30040
I20251025 20:09:24.728397 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:09:24.728624 18761 external_mini_cluster.cc:1468] Setting key fa7330b8e19d9cdc3e24c99a7e72214c
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:09:24.732774 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.723860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:44981 (local address 127.18.82.126:35985)
1025 20:09:24.724012 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.724017 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.724592 (+ 575us) server_negotiation.cc:408] Connection header received
1025 20:09:24.725268 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.725271 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.725321 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.725386 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:24.726552 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.727038 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.727593 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.727701 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.730189 (+ 2488us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:24.730199 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:24.730201 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:24.730223 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:24.731454 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.731835 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.731838 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.731839 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.731872 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.732056 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.732057 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.732058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.732170 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:24.732232 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.732587 (+ 355us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.732671 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 20:09:24.733325 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.724082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:35985 (local address 127.18.82.67:44981)
1025 20:09:24.724462 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.724505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.725114 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.725542 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.725552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.725895 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:09:24.726426 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.726438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.727124 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.727127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.727520 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.727526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.727668 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.728546 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:09:24.728567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:09:24.730071 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:09:24.731555 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.731558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.731571 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.731766 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.731927 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:09:24.731929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:09:24.731930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:09:24.731990 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:09:24.732232 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:09:24.732236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:09:24.732415 (+ 179us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.732557 (+ 142us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.732729 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20251025 20:09:24.734265 30169 heartbeater.cc:344] Connected to a master server at 127.18.82.126:35985
I20251025 20:09:24.734501 30169 heartbeater.cc:461] Registering TS with master...
I20251025 20:09:24.734912 30169 heartbeater.cc:507] Master 127.18.82.126:35985 requested a full tablet report, sending...
I20251025 20:09:24.735752 29704 ts_manager.cc:194] Registered new tserver with Master: 804f9b98d4244034a0ac45cee2b002f6 (127.18.82.67:43839)
I20251025 20:09:24.736268 29704 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:44981
I20251025 20:09:24.741954 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:09:24.748863 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.743535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.18.82.126:35985)
1025 20:09:24.743660 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.743663 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.743779 (+ 116us) server_negotiation.cc:408] Connection header received
1025 20:09:24.743859 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.743861 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.743893 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.743950 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:09:24.744644 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.745063 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.745631 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.745747 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.746732 (+ 985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:09:24.746745 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:09:24.746747 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:09:24.746763 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:09:24.747873 (+ 1110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.748168 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.748170 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.748171 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.748199 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:09:24.748362 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:09:24.748363 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:09:24.748365 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:09:24.748490 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:09:24.748576 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.748696 (+ 120us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.748769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 20:09:24.751084 29704 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:09:24.753211 29704 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:09:24.766454 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.760493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43899 (local address 127.0.0.1:57836)
1025 20:09:24.760860 (+ 367us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.760890 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.761008 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.763493 (+ 2485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.763496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.763514 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:09:24.763756 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.763762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.765068 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.765071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.765952 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.765963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.766054 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.766101 (+ 47us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.766194 (+ 93us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.766295 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20251025 20:09:24.767015 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.761438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:36425 (local address 127.0.0.1:56004)
1025 20:09:24.762491 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.762505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.762588 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.763667 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.763671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.763697 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:09:24.763961 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.763971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.765351 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.765356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.766359 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.766371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.766488 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.766795 (+ 307us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.766855 (+ 60us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.766910 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":109,"threads_started":1}
I20251025 20:09:24.767367 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.761864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.18.82.65:43899)
1025 20:09:24.763104 (+ 1240us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.763110 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.763129 (+ 19us) server_negotiation.cc:408] Connection header received
1025 20:09:24.763204 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.763208 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.763364 (+ 156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.763881 (+ 517us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:09:24.764002 (+ 121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.764955 (+ 953us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.766430 (+ 1475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.766980 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.767082 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.767145 (+ 63us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.767224 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":112,"threads_started":1}
I20251025 20:09:24.767747 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.761129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:43839 (local address 127.0.0.1:54214)
1025 20:09:24.762083 (+ 954us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.762137 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.762408 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.763098 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.763103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.763124 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:09:24.763364 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.763373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.765199 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.765202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.765983 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.765993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.766081 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.766101 (+ 20us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.766203 (+ 102us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.766294 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":77,"threads_started":1}
I20251025 20:09:24.767902 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.762174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.18.82.67:43839)
1025 20:09:24.762524 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.762532 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.762553 (+ 21us) server_negotiation.cc:408] Connection header received
1025 20:09:24.762636 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.762642 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.762813 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.762957 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:09:24.763494 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.765105 (+ 1611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.766637 (+ 1532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.767133 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.767215 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.767267 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.767331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":90,"threads_started":1}
I20251025 20:09:24.768226 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.761946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.18.82.66:36425)
1025 20:09:24.763081 (+ 1135us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.763089 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.763116 (+ 27us) server_negotiation.cc:408] Connection header received
1025 20:09:24.763202 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.763208 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.763379 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.763506 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:09:24.764110 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.765194 (+ 1084us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.767242 (+ 2048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.767700 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.767786 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.767998 (+ 212us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.768080 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":838,"threads_started":1}
I20251025 20:09:24.770694 29967 tablet_service.cc:1505] Processing CreateTablet for tablet 6a24c29e3b794373a61a096dbb55e694 (DEFAULT_TABLE table=test-table [id=1803b0363e7d49e687d10cb2a5d45884]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:09:24.770638 29831 tablet_service.cc:1505] Processing CreateTablet for tablet 6a24c29e3b794373a61a096dbb55e694 (DEFAULT_TABLE table=test-table [id=1803b0363e7d49e687d10cb2a5d45884]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:09:24.770634 30103 tablet_service.cc:1505] Processing CreateTablet for tablet 6a24c29e3b794373a61a096dbb55e694 (DEFAULT_TABLE table=test-table [id=1803b0363e7d49e687d10cb2a5d45884]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:09:24.771528 29967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a24c29e3b794373a61a096dbb55e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:24.771530 29831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a24c29e3b794373a61a096dbb55e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:24.771530 30103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a24c29e3b794373a61a096dbb55e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:09:24.776991 30186 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Bootstrap starting.
I20251025 20:09:24.778131 30188 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Bootstrap starting.
I20251025 20:09:24.779211 30186 tablet_bootstrap.cc:654] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:24.779771 30188 tablet_bootstrap.cc:654] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:24.779940 30186 log.cc:826] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:24.780417 30188 log.cc:826] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:24.780962 30187 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Bootstrap starting.
I20251025 20:09:24.781704 30186 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: No bootstrap required, opened a new log
I20251025 20:09:24.781891 30186 ts_tablet_manager.cc:1403] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:09:24.781931 30188 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: No bootstrap required, opened a new log
I20251025 20:09:24.782079 30188 ts_tablet_manager.cc:1403] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 20:09:24.782781 30187 tablet_bootstrap.cc:654] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Neither blocks nor log segments found. Creating new log.
I20251025 20:09:24.783459 30187 log.cc:826] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Log is configured to *not* fsync() on all Append() calls
I20251025 20:09:24.784958 30188 raft_consensus.cc:359] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.785178 30188 raft_consensus.cc:385] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:24.785241 30188 raft_consensus.cc:740] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804f9b98d4244034a0ac45cee2b002f6, State: Initialized, Role: FOLLOWER
I20251025 20:09:24.785390 30187 tablet_bootstrap.cc:492] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: No bootstrap required, opened a new log
I20251025 20:09:24.785390 30186 raft_consensus.cc:359] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.785645 30187 ts_tablet_manager.cc:1403] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 20:09:24.785661 30186 raft_consensus.cc:385] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:24.785710 30186 raft_consensus.cc:740] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020e0d109c3e451092094d9abebda679, State: Initialized, Role: FOLLOWER
I20251025 20:09:24.785724 30188 consensus_queue.cc:260] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.786149 30186 consensus_queue.cc:260] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.786564 30188 ts_tablet_manager.cc:1434] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251025 20:09:24.786862 30169 heartbeater.cc:499] Master 127.18.82.126:35985 was elected leader, sending a full tablet report...
I20251025 20:09:24.786867 30033 heartbeater.cc:499] Master 127.18.82.126:35985 was elected leader, sending a full tablet report...
I20251025 20:09:24.788007 30186 ts_tablet_manager.cc:1434] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251025 20:09:24.788626 30187 raft_consensus.cc:359] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.788753 30187 raft_consensus.cc:385] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:09:24.788781 30187 raft_consensus.cc:740] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa75d2b46da4c56951a38cdf7f008d0, State: Initialized, Role: FOLLOWER
I20251025 20:09:24.789054 30187 consensus_queue.cc:260] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.789665 29897 heartbeater.cc:499] Master 127.18.82.126:35985 was elected leader, sending a full tablet report...
I20251025 20:09:24.789752 30187 ts_tablet_manager.cc:1434] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20251025 20:09:24.811986 29898 tablet.cc:2378] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:09:24.852128 30194 raft_consensus.cc:493] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:09:24.852301 30194 raft_consensus.cc:515] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.853259 30194 leader_election.cc:290] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804f9b98d4244034a0ac45cee2b002f6 (127.18.82.67:43839), 020e0d109c3e451092094d9abebda679 (127.18.82.66:36425)
I20251025 20:09:24.856099 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.853442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:43839 (local address 127.18.82.65:34055)
1025 20:09:24.853704 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.853720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.853822 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.854174 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.854177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.854197 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:09:24.854451 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.854458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.855156 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.855158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.855716 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.855724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.855840 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.855861 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.855910 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.855963 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20251025 20:09:24.856595 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.853575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:34055 (local address 127.18.82.67:43839)
1025 20:09:24.853701 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.853704 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.853793 (+ 89us) server_negotiation.cc:408] Connection header received
1025 20:09:24.854010 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.854016 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.854076 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.854155 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:09:24.854573 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.855040 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.855843 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.856312 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.856350 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.856407 (+ 57us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.856470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 20:09:24.856622 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.853815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:36425 (local address 127.18.82.65:53303)
1025 20:09:24.854116 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.854126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 20:09:24.854243 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:09:24.854501 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:09:24.854504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:09:24.854521 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:09:24.854756 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.854762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.855422 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.855425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:09:24.855969 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:09:24.855976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.856076 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.856094 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:09:24.856239 (+ 145us) client_negotiation.cc:240] Negotiation successful
1025 20:09:24.856298 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":11392,"thread_start_us":119,"threads_started":1}
I20251025 20:09:24.856868 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.853952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:53303 (local address 127.18.82.66:36425)
1025 20:09:24.854100 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.854106 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.854199 (+ 93us) server_negotiation.cc:408] Connection header received
1025 20:09:24.854343 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.854346 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.854394 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.854483 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:09:24.854866 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.855317 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.856082 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.856584 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.856623 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.856686 (+ 63us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.856747 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251025 20:09:24.857138 30123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a24c29e3b794373a61a096dbb55e694" candidate_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804f9b98d4244034a0ac45cee2b002f6" is_pre_election: true
I20251025 20:09:24.857451 30123 raft_consensus.cc:2468] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa75d2b46da4c56951a38cdf7f008d0 in term 0.
I20251025 20:09:24.857678 29987 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a24c29e3b794373a61a096dbb55e694" candidate_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "020e0d109c3e451092094d9abebda679" is_pre_election: true
I20251025 20:09:24.857888 29987 raft_consensus.cc:2468] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa75d2b46da4c56951a38cdf7f008d0 in term 0.
I20251025 20:09:24.857911 29787 leader_election.cc:304] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa75d2b46da4c56951a38cdf7f008d0, 804f9b98d4244034a0ac45cee2b002f6; no voters:
I20251025 20:09:24.858103 30194 raft_consensus.cc:2804] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:09:24.858178 30194 raft_consensus.cc:493] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:09:24.858222 30194 raft_consensus.cc:3060] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:24.859122 30194 raft_consensus.cc:515] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.859443 30194 leader_election.cc:290] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [CANDIDATE]: Term 1 election: Requested vote from peers 804f9b98d4244034a0ac45cee2b002f6 (127.18.82.67:43839), 020e0d109c3e451092094d9abebda679 (127.18.82.66:36425)
I20251025 20:09:24.859738 30123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a24c29e3b794373a61a096dbb55e694" candidate_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804f9b98d4244034a0ac45cee2b002f6"
I20251025 20:09:24.859835 29987 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a24c29e3b794373a61a096dbb55e694" candidate_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "020e0d109c3e451092094d9abebda679"
I20251025 20:09:24.859875 30123 raft_consensus.cc:3060] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:24.859948 29987 raft_consensus.cc:3060] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:09:24.861030 30123 raft_consensus.cc:2468] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa75d2b46da4c56951a38cdf7f008d0 in term 1.
I20251025 20:09:24.861068 29987 raft_consensus.cc:2468] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa75d2b46da4c56951a38cdf7f008d0 in term 1.
I20251025 20:09:24.861299 29787 leader_election.cc:304] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 020e0d109c3e451092094d9abebda679, 2fa75d2b46da4c56951a38cdf7f008d0; no voters:
I20251025 20:09:24.861440 30194 raft_consensus.cc:2804] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:09:24.861627 30194 raft_consensus.cc:697] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa75d2b46da4c56951a38cdf7f008d0, State: Running, Role: LEADER
I20251025 20:09:24.861889 30194 consensus_queue.cc:237] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } }
I20251025 20:09:24.864509 29703 catalog_manager.cc:5649] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa75d2b46da4c56951a38cdf7f008d0 (127.18.82.65). New cstate: current_term: 1 leader_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa75d2b46da4c56951a38cdf7f008d0" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 43899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 } health_report { overall_health: UNKNOWN } } }
I20251025 20:09:24.887228 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.884420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.18.82.65:43899)
1025 20:09:24.884528 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1025 20:09:24.884532 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:09:24.884542 (+ 10us) server_negotiation.cc:408] Connection header received
1025 20:09:24.884690 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:09:24.884693 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:09:24.884739 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:09:24.884829 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:09:24.885151 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.885609 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:09:24.886185 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:09:24.886394 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:09:24.886478 (+ 84us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:09:24.886845 (+ 367us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:09:24.886955 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 20:09:24.887062 (+ 107us) server_negotiation.cc:300] Negotiation successful
1025 20:09:24.887108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251025 20:09:24.891999 30123 raft_consensus.cc:1275] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 1 FOLLOWER]: Refusing update from remote peer 2fa75d2b46da4c56951a38cdf7f008d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:09:24.892000 29987 raft_consensus.cc:1275] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 1 FOLLOWER]: Refusing update from remote peer 2fa75d2b46da4c56951a38cdf7f008d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:09:24.892550 30194 consensus_queue.cc:1048] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "020e0d109c3e451092094d9abebda679" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:09:24.892680 30197 consensus_queue.cc:1048] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "804f9b98d4244034a0ac45cee2b002f6" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 43839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:09:24.901196 30202 mvcc.cc:204] Tried to move back new op lower bound from 7214788464191934464 to 7214788464076656640. Current Snapshot: MvccSnapshot[applied={T|T < 7214788464191934464}]
I20251025 20:09:24.906117 29702 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:09:24.906263 29702 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:09:24.908380 29702 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 361de9900136414892dbfac131963848: Sending DeleteTablet for 3 replicas of tablet 6a24c29e3b794373a61a096dbb55e694
I20251025 20:09:24.909039 29831 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a24c29e3b794373a61a096dbb55e694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:09:24 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:57836
I20251025 20:09:24.909235 29967 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a24c29e3b794373a61a096dbb55e694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:09:24 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:56004
I20251025 20:09:24.909245 30103 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a24c29e3b794373a61a096dbb55e694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:09:24 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:54214
I20251025 20:09:24.909543 30210 tablet_replica.cc:333] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0: stopping tablet replica
I20251025 20:09:24.909657 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29768
I20251025 20:09:24.909790 30210 raft_consensus.cc:2243] T 6a24c29e3b794373a61a096dbb55e694 P 2fa75d2b46da4c56951a38cdf7f008d0 [term 1 LEADER]: Raft consensus shutting down.
I20251025 20:09:24.910590 30211 tablet_replica.cc:333] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: stopping tablet replica
I20251025 20:09:24.910856 30211 raft_consensus.cc:2243] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:09:24.911131 30211 raft_consensus.cc:2272] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:09:24.911404 30212 tablet_replica.cc:333] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: stopping tablet replica
I20251025 20:09:24.911607 30212 raft_consensus.cc:2243] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:09:24.911842 30212 raft_consensus.cc:2272] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:09:24.913022 30212 ts_tablet_manager.cc:1916] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:09:24.913429 30211 ts_tablet_manager.cc:1916] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:09:24.915004 30212 ts_tablet_manager.cc:1929] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:09:24.915081 30212 log.cc:1199] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/6a24c29e3b794373a61a096dbb55e694
I20251025 20:09:24.915364 30212 ts_tablet_manager.cc:1950] T 6a24c29e3b794373a61a096dbb55e694 P 804f9b98d4244034a0ac45cee2b002f6: Deleting consensus metadata
I20251025 20:09:24.916265 29691 catalog_manager.cc:4985] TS 804f9b98d4244034a0ac45cee2b002f6 (127.18.82.67:43839): tablet 6a24c29e3b794373a61a096dbb55e694 (table test-table [id=1803b0363e7d49e687d10cb2a5d45884]) successfully deleted
I20251025 20:09:24.916402 30211 ts_tablet_manager.cc:1929] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:09:24.916494 30211 log.cc:1199] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/6a24c29e3b794373a61a096dbb55e694
I20251025 20:09:24.916838 30211 ts_tablet_manager.cc:1950] T 6a24c29e3b794373a61a096dbb55e694 P 020e0d109c3e451092094d9abebda679: Deleting consensus metadata
I20251025 20:09:24.917888 29691 catalog_manager.cc:4985] TS 020e0d109c3e451092094d9abebda679 (127.18.82.66:36425): tablet 6a24c29e3b794373a61a096dbb55e694 (table test-table [id=1803b0363e7d49e687d10cb2a5d45884]) successfully deleted
W20251025 20:09:24.918057 29690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:43899 (error 108)
I20251025 20:09:24.918140 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29904
I20251025 20:09:24.919296 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:09:24.918987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:43899 (local address 127.0.0.1:57856)
1025 20:09:24.919137 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1025 20:09:24.919207 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20251025 20:09:24.919436 29690 catalog_manager.cc:4712] TS 2fa75d2b46da4c56951a38cdf7f008d0 (127.18.82.65:43899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a24c29e3b794373a61a096dbb55e694: Network error: Client connection negotiation failed: client connection to 127.18.82.65:43899: connect: Connection refused (error 111)
I20251025 20:09:24.924598 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 30040
I20251025 20:09:24.930006 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 29672
2025-10-25T20:09:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2965 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[30216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[30216](info): set up 2 sockets
Oct 25 20:09:24 dist-test-slave-d2pk krb5kdc[30216](info): commencing operation
krb5kdc: starting...
W20251025 20:09:26.753830 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.803s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:09:26 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761422966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:09:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:09:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangeradmin_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangeradmin_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangerlookup_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangerlookup_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
I20251025 20:09:26.889432 18761 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKRacCB/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/postgres -l logfile start
2025-10-25 20:09:29.089 UTC [30247] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-25 20:09:29.089 UTC [30247] LOG: listening on IPv4 address "127.18.82.84", port 54725
2025-10-25 20:09:29.092 UTC [30247] LOG: listening on Unix socket "/tmp/.s.PGSQL.54725"
2025-10-25 20:09:29.096 UTC [30252] LOG: database system was shut down at 2025-10-25 20:09:27 UTC
2025-10-25 20:09:29.100 UTC [30247] LOG: database system is ready to accept connections
I20251025 20:09:30.897846 18761 mini_postgres.cc:96] Postgres bound to 54725
2025-10-25 20:09:30.903 UTC [30258] FATAL: database "slave" does not exist
127.18.82.84:54725 - accepting connections
I20251025 20:09:30.904222 18761 mini_ranger.cc:161] Starting Ranger...
I20251025 20:09:30.920795 18761 mini_ranger.cc:84] Created miniranger Postgres user
I20251025 20:09:30.984668 18761 mini_ranger.cc:87] Created ranger Postgres database
I20251025 20:09:30.984761 18761 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin
2025-10-25 20:09:31,533 [I] DB FLAVOR :POSTGRES
2025-10-25 20:09:31,534 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 20:09:31,534 [I] Checking connection..
2025-10-25 20:09:31,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-25 20:09:31,881 [I] Checking connection passed.
2025-10-25 20:09:31,881 [I] --------- Verifying version history table ---------
2025-10-25 20:09:31,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:09:32,156 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 20:09:32,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:09:32,440 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 20:09:32,440 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-25 20:09:32,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-25 20:09:32.733 UTC [30325] WARNING: there is no transaction in progress
2025-10-25 20:09:32,739 [I] create_dbversion_catalog.sql file imported successfully
2025-10-25 20:09:32,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:09:33,055 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-25 20:09:33,055 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 20:09:33,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-25 20:09:33,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-25 20:09:33,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d2pk.c.gcp-upstream.internal','N') ;"
2025-10-25 20:09:33,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 20:09:34,236 [I] Table x_portal_user does not exist in database ranger
2025-10-25 20:09:34,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 20:09:34,522 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-25 20:09:34,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 20:09:34,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 20:09:35,086 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-25 20:09:35,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-25 20:09:35.848 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.860 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.871 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.881 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.890 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.901 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.941 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.945 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.953 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.961 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.968 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.977 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.983 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.987 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.990 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:35.993 UTC [30442] WARNING: there is no transaction in progress
2025-10-25 20:09:36,336 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-25 20:09:36,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-25 20:09:36,612 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-25 20:09:36,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 20:09:36,918 [I] Table x_portal_user already exists in database 'ranger'
2025-10-25 20:09:36,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 20:09:37,221 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-25 20:09:37,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 20:09:37,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 20:09:37,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d2pk.c.gcp-upstream.internal';"
2025-10-25 20:09:38,057 [I] CORE_DB_SCHEMA import status has been updated
2025-10-25 20:09:38,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-25 20:09:38,360 [I] DB_PATCHES have already been applied
I20251025 20:09:38.365221 18761 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKRacCB/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251025 20:09:38.365278 18761 mini_ranger.cc:193] Using host: 127.18.82.84
I20251025 20:09:38.368249 18761 mini_ranger.cc:227] Ranger admin URL: http://127.18.82.84:32887
Oct 25, 2025 8:09:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:09:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:09:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp
Oct 25, 2025 8:09:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 25, 2025 8:09:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 25, 2025 8:09:39 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 25, 2025 8:09:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp].
Oct 25, 2025 8:09:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 25, 2025 8:09:39 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-32887"]
Oct 25, 2025 8:09:39 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:09:39 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:09:39 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 25, 2025 8:09:39 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 25, 2025 8:09:39 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251025 20:09:40.418931 18761 mini_ranger.cc:160] Time spent starting Ranger: real 9.515s user 0.002s sys 0.007s
Oct 25, 2025 8:09:40 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 25, 2025 8:09:40 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25, 2025 8:09:40 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 25, 2025 8:09:40 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 25, 2025 8:09:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 25, 2025 8:09:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Oct 25, 2025 8:09:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 25, 2025 8:09:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 25, 2025 8:09:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 25, 2025 8:09:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-32887"]
I20251025 20:10:00.251341 18761 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangerkms_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/rangerkms_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.82.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.82.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 25 20:10:00 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423000, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251025 20:10:00.316473 18761 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251025 20:10:00.325263 18761 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251025 20:10:00.426851 18761 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251025 20:10:00.426959 18761 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms
I20251025 20:10:00.426998 18761 mini_ranger_kms.cc:227] Using postgres at 127.18.82.84:54725
2025-10-25 20:10:01,048 [I] DB FLAVOR :POSTGRES
2025-10-25 20:10:01,049 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 20:10:01,049 [I] Checking connection
2025-10-25 20:10:01,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-25 20:10:01,349 [I] connection success
2025-10-25 20:10:01,350 [I] --------- Verifying Ranger DB tables ---------
2025-10-25 20:10:01,350 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-25 20:10:01,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-25 20:10:01,652 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-25 20:10:01,653 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 20:10:01,653 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-25 20:10:01,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:54725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-25 20:10:01,938 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251025 20:10:02.366901 18761 mini_ranger_kms.cc:310] Created kms service
I20251025 20:10:02.560881 18761 mini_ranger_kms.cc:326] Created kudu user
I20251025 20:10:02.599869 18761 mini_ranger_kms.cc:343] Created rangerkms user
Oct 25 20:10:02 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423000, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:02.989174 18761 mini_ranger_kms.cc:384] Added ranger policy
I20251025 20:10:02.989383 18761 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251025 20:10:02.989408 18761 mini_ranger_kms.cc:240] Using host: 127.18.82.84
I20251025 20:10:02.993425 18761 mini_ranger_kms.cc:276] Ranger KMS PID: 30634
I20251025 20:10:02.993501 18761 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.82.84:59145
Oct 25, 2025 8:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 25, 2025 8:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 25, 2025 8:10:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp].
Oct 25, 2025 8:10:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 25, 2025 8:10:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-59145"]
Oct 25, 2025 8:10:04 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:10:04 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:10:04 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 25, 2025 8:10:04 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25 20:10:04 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761423004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:05.018392 18761 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.702s user 0.000s sys 0.229s
I20251025 20:10:05.018522 18761 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251025 20:10:05.018610 18761 mini_ranger_kms.cc:401] 127.18.82.84:59145/kms/v1/keys
Oct 25 20:10:05 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
Oct 25 20:10:05 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
Oct 25 20:10:07 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:07.714468 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:38381
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:38381
--ranger_config_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:59145/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:07.813338 30687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:07.813580 30687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:07.813685 30687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:07.816473 30687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:10:07.816529 30687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:07.816550 30687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:07.816568 30687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:10:07.816603 30687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:10:07.820705 30687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.82.84:59145/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:38381
--ranger_config_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:07.821657 30687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:07.822422 30687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:07.828022 30692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:07.828064 30693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:07.828202 30695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:07.828387 30687 server_base.cc:1047] running on GCE node
I20251025 20:10:07.828788 30687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:07.829610 30687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:07.830754 30687 hybrid_clock.cc:648] HybridClock initialized: now 1761423007830741 us; error 31 us; skew 500 ppm
Oct 25 20:10:07 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423007, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:07.833364 30687 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:10:07.834336 30687 webserver.cc:492] Webserver started at http://127.18.82.126:41413/ using document root <none> and password file <none>
I20251025 20:10:07.834817 30687 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:07.834867 30687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:07.835028 30687 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:07 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423007, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:07.875941 30687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b"
format_stamp: "Formatted at 2025-10-25 20:10:07 on dist-test-slave-d2pk"
server_key: "8f59ac646dfe15d079955e524b728218"
server_key_iv: "01f1852f89f272bd5bab8b23e13bfd08"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:07.876463 30687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b"
format_stamp: "Formatted at 2025-10-25 20:10:07 on dist-test-slave-d2pk"
server_key: "8f59ac646dfe15d079955e524b728218"
server_key_iv: "01f1852f89f272bd5bab8b23e13bfd08"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:07.880430 30687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20251025 20:10:07.892239 30702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:07.893283 30687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 20:10:07.893388 30687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b"
format_stamp: "Formatted at 2025-10-25 20:10:07 on dist-test-slave-d2pk"
server_key: "8f59ac646dfe15d079955e524b728218"
server_key_iv: "01f1852f89f272bd5bab8b23e13bfd08"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:07.893487 30687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:07.904639 30687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:07.911633 30687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:07.911823 30687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:07.919862 30687 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:38381
I20251025 20:10:07.919893 30754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:38381 every 8 connection(s)
I20251025 20:10:07.920887 30687 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:10:07.924125 30755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:10:07.929939 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 30687
I20251025 20:10:07.930060 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:10:07.930167 30755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: Bootstrap starting.
I20251025 20:10:07.930323 18761 external_mini_cluster.cc:1468] Setting key a573864e47d43ffa53bf74786158a832
I20251025 20:10:07.932724 30755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: Neither blocks nor log segments found. Creating new log.
I20251025 20:10:07.933590 30755 log.cc:826] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: Log is configured to *not* fsync() on all Append() calls
I20251025 20:10:07.935313 30755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: No bootstrap required, opened a new log
Oct 25 20:10:07 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:10:07.938887 30755 raft_consensus.cc:359] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } }
I20251025 20:10:07.939111 30755 raft_consensus.cc:385] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:10:07.939185 30755 raft_consensus.cc:740] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06bea56e66246c0b5e9ccbcdacb2a8b, State: Initialized, Role: FOLLOWER
I20251025 20:10:07.939643 30755 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } }
I20251025 20:10:07.939797 30755 raft_consensus.cc:399] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:10:07.939870 30755 raft_consensus.cc:493] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:10:07.939986 30755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:10:07.940758 30755 raft_consensus.cc:515] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } }
I20251025 20:10:07.941121 30755 leader_election.cc:304] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06bea56e66246c0b5e9ccbcdacb2a8b; no voters:
I20251025 20:10:07.941485 30755 leader_election.cc:290] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:10:07.941591 30760 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:10:07.942037 30760 raft_consensus.cc:697] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [term 1 LEADER]: Becoming Leader. State: Replica: b06bea56e66246c0b5e9ccbcdacb2a8b, State: Running, Role: LEADER
I20251025 20:10:07.942358 30760 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } }
I20251025 20:10:07.942718 30755 sys_catalog.cc:565] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:10:07.943476 30760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } } }
I20251025 20:10:07.943660 30760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [sys.catalog]: This master's current role is: LEADER
I20251025 20:10:07.943879 30760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06bea56e66246c0b5e9ccbcdacb2a8b. Latest consensus state: current_term: 1 leader_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bea56e66246c0b5e9ccbcdacb2a8b" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 38381 } } }
I20251025 20:10:07.944011 30760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b [sys.catalog]: This master's current role is: LEADER
I20251025 20:10:07.944057 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:07.931542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.18.82.126:38381)
1025 20:10:07.931945 (+ 403us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:07.931954 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:07.931986 (+ 32us) server_negotiation.cc:408] Connection header received
1025 20:10:07.932455 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:07.932471 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:07.932762 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:07.933080 (+ 318us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:07.934092 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:07.935258 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:07.936029 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:07.936372 (+ 343us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:07.938814 (+ 2442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:07.938832 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:07.938844 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:07.938885 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:07.940966 (+ 2081us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:07.941431 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:07.941436 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:07.941441 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:07.941553 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:07.941865 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:07.941868 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:07.941870 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:07.942258 (+ 388us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:07.943038 (+ 780us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:07.943186 (+ 148us) server_negotiation.cc:300] Negotiation successful
1025 20:10:07.943386 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20251025 20:10:07.944324 30769 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:10:07.948184 30769 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:10:07.948670 30755 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251025 20:10:07.957425 30769 catalog_manager.cc:1357] Generated new cluster ID: ca7c204f2af240548365d813e5c95923
I20251025 20:10:07.957593 30769 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:10:07.993791 30769 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:10:07.994993 30769 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:10:08.013355 30769 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: Generated new TSK 0
I20251025 20:10:08.014370 30769 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:08.840950 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:59145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:38381
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:08.966609 30801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:08.966807 30801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:08.966888 30801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:08.969843 30801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:08.969895 30801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:08.969980 30801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:10:08.973857 30801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.82.84:59145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:08.974824 30801 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:08.975623 30801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:08.981714 30807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:08.981750 30806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:08.981760 30809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:08.982164 30801 server_base.cc:1047] running on GCE node
I20251025 20:10:08.982465 30801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:08.982931 30801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:08.984103 30801 hybrid_clock.cc:648] HybridClock initialized: now 1761423008984081 us; error 38 us; skew 500 ppm
Oct 25 20:10:08 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:08.986876 30801 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:10:08.987991 30801 webserver.cc:492] Webserver started at http://127.18.82.65:33693/ using document root <none> and password file <none>
I20251025 20:10:08.988613 30801 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:08.988713 30801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:08.988912 30801 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:08 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:09.002736 30801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "9d78f7a2f81a435c992d2cb98e10fada"
format_stamp: "Formatted at 2025-10-25 20:10:08 on dist-test-slave-d2pk"
server_key: "6a22dd31b750041d0ecb0025d29e5114"
server_key_iv: "6d2ba06ff28b48002970456a51d89430"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.003211 30801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "9d78f7a2f81a435c992d2cb98e10fada"
format_stamp: "Formatted at 2025-10-25 20:10:08 on dist-test-slave-d2pk"
server_key: "6a22dd31b750041d0ecb0025d29e5114"
server_key_iv: "6d2ba06ff28b48002970456a51d89430"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.006533 30801 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 20:10:09.020128 30816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.021490 30801 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:10:09.021705 30801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "9d78f7a2f81a435c992d2cb98e10fada"
format_stamp: "Formatted at 2025-10-25 20:10:08 on dist-test-slave-d2pk"
server_key: "6a22dd31b750041d0ecb0025d29e5114"
server_key_iv: "6d2ba06ff28b48002970456a51d89430"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.022004 30801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:09.051389 30801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:09.054563 30801 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:09.054744 30801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:09.055367 30801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:10:09.056393 30801 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:10:09.056438 30801 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.056486 30801 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:10:09.056533 30801 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.067039 30801 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:45135
I20251025 20:10:09.067260 30931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:45135 every 8 connection(s)
I20251025 20:10:09.068080 30801 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:10:09.076663 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 30801
I20251025 20:10:09.076792 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:10:09.077076 18761 external_mini_cluster.cc:1468] Setting key 4008f71b9d7a2e3724e12a0ff8b47b3e
I20251025 20:10:09.079797 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.069915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:47769 (local address 127.18.82.126:38381)
1025 20:10:09.070185 (+ 270us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.070189 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.070719 (+ 530us) server_negotiation.cc:408] Connection header received
1025 20:10:09.071607 (+ 888us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.071611 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.071675 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.071765 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:09.072891 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.073428 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.074060 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.074215 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.076211 (+ 1996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:09.076228 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:09.076231 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:09.076253 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:09.077927 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.078404 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.078407 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.078411 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.078452 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.078755 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.078758 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.078759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.078887 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:09.078962 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.079496 (+ 534us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.079599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20251025 20:10:09.080539 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.070173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:38381 (local address 127.18.82.65:47769)
1025 20:10:09.070540 (+ 367us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.070582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.071370 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.071947 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.071954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.072192 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:10:09.072753 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.072765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.073570 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.073573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.073955 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.073963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.074157 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.074762 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:10:09.074783 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:10:09.076074 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:10:09.078028 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.078033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.078043 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.078312 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.078533 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.078535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.078537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.078656 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.078988 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:10:09.078996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:10:09.079245 (+ 249us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.079454 (+ 209us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.079699 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
I20251025 20:10:09.081645 30932 heartbeater.cc:344] Connected to a master server at 127.18.82.126:38381
I20251025 20:10:09.081900 30932 heartbeater.cc:461] Registering TS with master...
I20251025 20:10:09.082398 30932 heartbeater.cc:507] Master 127.18.82.126:38381 requested a full tablet report, sending...
I20251025 20:10:09.083770 30719 ts_manager.cc:194] Registered new tserver with Master: 9d78f7a2f81a435c992d2cb98e10fada (127.18.82.65:45135)
I20251025 20:10:09.084833 30719 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:47769
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:09.127847 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:59145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:38381
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:09.248390 30940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:09.248752 30940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:09.248870 30940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:09.252640 30940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:09.252704 30940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:09.252818 30940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:10:09.256947 30940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.82.84:59145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:09.257964 30940 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:09.258930 30940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20251025 20:10:09.265631 30950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:09.265753 30947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:09.265702 30948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:09.266350 30940 server_base.cc:1047] running on GCE node
I20251025 20:10:09.266861 30940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:09.267454 30940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:09.268642 30940 hybrid_clock.cc:648] HybridClock initialized: now 1761423009268622 us; error 31 us; skew 500 ppm
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:09.271073 30940 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:10:09.272202 30940 webserver.cc:492] Webserver started at http://127.18.82.66:39867/ using document root <none> and password file <none>
I20251025 20:10:09.273126 30940 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:09.273219 30940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:09.273458 30940 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:09.289469 30940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "c1d11b8a50f842708287978eaf950b85"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "cbf43e6bdcf2bce70858de8af52e3434"
server_key_iv: "1712a98d3aec4d6b8ab4668f970cfd0f"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.290122 30940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "c1d11b8a50f842708287978eaf950b85"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "cbf43e6bdcf2bce70858de8af52e3434"
server_key_iv: "1712a98d3aec4d6b8ab4668f970cfd0f"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.296077 30940 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20251025 20:10:09.323851 30957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.325129 30940 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.000s sys 0.002s
I20251025 20:10:09.325253 30940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "c1d11b8a50f842708287978eaf950b85"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "cbf43e6bdcf2bce70858de8af52e3434"
server_key_iv: "1712a98d3aec4d6b8ab4668f970cfd0f"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.325373 30940 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:09.356181 30940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:09.359491 30940 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:09.359685 30940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:09.360311 30940 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:10:09.361356 30940 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:10:09.361408 30940 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.361466 30940 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:10:09.361495 30940 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.373704 30940 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:35255
I20251025 20:10:09.374891 30940 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:10:09.375720 31071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:35255 every 8 connection(s)
I20251025 20:10:09.384513 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 30940
I20251025 20:10:09.384655 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:10:09.384940 18761 external_mini_cluster.cc:1468] Setting key e1de1441f6d896cd2272f4a0df041e1e
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.82.67@KRBTEST.COM; defaulting to no policy
I20251025 20:10:09.398618 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.378896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.66:54645 (local address 127.18.82.126:38381)
1025 20:10:09.379042 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.379048 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.379064 (+ 16us) server_negotiation.cc:408] Connection header received
1025 20:10:09.380652 (+ 1588us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.380656 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.380733 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.380812 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:09.382793 (+ 1981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.383581 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.384424 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.384617 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.388025 (+ 3408us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:09.388043 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:09.388046 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:09.388071 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:09.390218 (+ 2147us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.394092 (+ 3874us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.394098 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.394101 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.394159 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.397322 (+ 3163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.397325 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.397327 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.397534 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:09.397639 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.398327 (+ 688us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.398449 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 20:10:09.400130 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.377882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:38381 (local address 127.18.82.66:54645)
1025 20:10:09.378372 (+ 490us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.378414 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.380356 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.381484 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.381495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.381885 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:10:09.382636 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.382653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.383716 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.383719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.384275 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.384286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.384829 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.385697 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:10:09.385726 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:10:09.387838 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:10:09.393608 (+ 5770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.393613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.393625 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.393981 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.397108 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.397111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.397112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.397219 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.397715 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:10:09.397722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:10:09.398042 (+ 320us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.398993 (+ 951us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.399253 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.67@KRBTEST.COM" created.
I20251025 20:10:09.401603 31072 heartbeater.cc:344] Connected to a master server at 127.18.82.126:38381
I20251025 20:10:09.401867 31072 heartbeater.cc:461] Registering TS with master...
I20251025 20:10:09.402532 31072 heartbeater.cc:507] Master 127.18.82.126:38381 requested a full tablet report, sending...
I20251025 20:10:09.403592 30719 ts_manager.cc:194] Registered new tserver with Master: c1d11b8a50f842708287978eaf950b85 (127.18.82.66:35255)
I20251025 20:10:09.404318 30719 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.66@KRBTEST.COM'} at 127.18.82.66:54645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:09.438628 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:59145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.67:0
--local_ip_for_outbound_sockets=127.18.82.67
--webserver_interface=127.18.82.67
--webserver_port=0
--tserver_master_addrs=127.18.82.126:38381
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:09.571626 31079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:09.571875 31079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:09.571974 31079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:09.584578 31079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:09.584681 31079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:09.584806 31079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.67
I20251025 20:10:09.589695 31079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.82.84:59145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.82.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.67
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:09.590970 31079 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:09.591852 31079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:09.602828 31084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:09.603040 31085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:09.603896 31087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:09.604722 31079 server_base.cc:1047] running on GCE node
I20251025 20:10:09.605185 31079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:09.605911 31079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:09.609216 31079 hybrid_clock.cc:648] HybridClock initialized: now 1761423009609162 us; error 67 us; skew 500 ppm
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:09.612283 31079 init.cc:377] Logged in from keytab as kudu/127.18.82.67@KRBTEST.COM (short username kudu)
I20251025 20:10:09.613488 31079 webserver.cc:492] Webserver started at http://127.18.82.67:42079/ using document root <none> and password file <none>
I20251025 20:10:09.614279 31079 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:09.614398 31079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:09.614632 31079 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:09.628495 31079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data/instance:
uuid: "397b718c2e9e4debae25e55083d685e7"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "5b498182b40773ed35b9702f74e4bc0a"
server_key_iv: "362f99305d454a719af108e13948e1d9"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.629038 31079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance:
uuid: "397b718c2e9e4debae25e55083d685e7"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "5b498182b40773ed35b9702f74e4bc0a"
server_key_iv: "362f99305d454a719af108e13948e1d9"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.633811 31079 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251025 20:10:09.662171 31094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.663415 31079 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251025 20:10:09.663535 31079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal
uuid: "397b718c2e9e4debae25e55083d685e7"
format_stamp: "Formatted at 2025-10-25 20:10:09 on dist-test-slave-d2pk"
server_key: "5b498182b40773ed35b9702f74e4bc0a"
server_key_iv: "362f99305d454a719af108e13948e1d9"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:09.663656 31079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:09.707718 31079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:09.716277 31079 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:09.716468 31079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:09.717080 31079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:10:09.721624 31079 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:10:09.721694 31079 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.721756 31079 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:10:09.721789 31079 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:09.733518 31079 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.67:46227
I20251025 20:10:09.734836 31079 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/data/info.pb
I20251025 20:10:09.736083 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 31079
I20251025 20:10:09.736219 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal/instance
I20251025 20:10:09.736505 18761 external_mini_cluster.cc:1468] Setting key 7163aba89e2d59c71f935a055ece9620
I20251025 20:10:09.738708 31207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.67:46227 every 8 connection(s)
Oct 25 20:10:09 dist-test-slave-d2pk krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.67@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:10:09.751175 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.742350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.67:36419 (local address 127.18.82.126:38381)
1025 20:10:09.742478 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.742483 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.742497 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:10:09.743253 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.743257 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.743314 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.743409 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:09.744649 (+ 1240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.745115 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.745783 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.745942 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.747966 (+ 2024us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:09.747978 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:09.747981 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:09.748003 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:09.749338 (+ 1335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.749821 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.749823 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.749825 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.749865 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.750187 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.750190 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.750191 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.750319 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:09.750399 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.750906 (+ 507us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.751023 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 20:10:09.752058 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.741309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:38381 (local address 127.18.82.67:36419)
1025 20:10:09.742126 (+ 817us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.742171 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.743059 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.743560 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.743568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.743824 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:10:09.744473 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.744488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.745228 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.745231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.745654 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.745662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.745860 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.746606 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:10:09.746630 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:10:09.747806 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:10:09.749440 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.749445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.749455 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.749721 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.749950 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:09.749955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:09.749957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:09.750081 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:09.750411 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:10:09.750419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:10:09.750634 (+ 215us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.750845 (+ 211us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.751065 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":149,"threads_started":1}
I20251025 20:10:09.753105 31208 heartbeater.cc:344] Connected to a master server at 127.18.82.126:38381
I20251025 20:10:09.753335 31208 heartbeater.cc:461] Registering TS with master...
I20251025 20:10:09.753866 31208 heartbeater.cc:507] Master 127.18.82.126:38381 requested a full tablet report, sending...
I20251025 20:10:09.754843 30718 ts_manager.cc:194] Registered new tserver with Master: 397b718c2e9e4debae25e55083d685e7 (127.18.82.67:46227)
I20251025 20:10:09.755370 30718 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.67@KRBTEST.COM'} at 127.18.82.67:36419
I20251025 20:10:09.764345 18761 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 20:10:09.772552 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.766370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.18.82.126:38381)
1025 20:10:09.766508 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.766512 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.766676 (+ 164us) server_negotiation.cc:408] Connection header received
1025 20:10:09.766784 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.766786 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.766824 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.766879 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:09.767675 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.768249 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.768783 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.768948 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.769976 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:09.769990 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:09.769993 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:09.770017 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:09.771398 (+ 1381us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.771735 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.771739 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.771741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.771784 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:09.771964 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:09.771967 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:09.771968 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:09.772101 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:09.772205 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.772345 (+ 140us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.772437 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 20:10:09.775012 30718 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 20:10:09.778194 30718 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 20:10:09.792760 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.787802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:46227 (local address 127.0.0.1:40734)
1025 20:10:09.789650 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.789666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.789778 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.790245 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.790250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.790269 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:10:09.790485 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.790490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.791505 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.791509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.792263 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.792272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.792377 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.792432 (+ 55us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.792531 (+ 99us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.792617 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"spinlock_wait_cycles":6016,"thread_start_us":67,"threads_started":1}
I20251025 20:10:09.793362 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.788425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.18.82.67:46227)
1025 20:10:09.789071 (+ 646us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.789078 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.789762 (+ 684us) server_negotiation.cc:408] Connection header received
1025 20:10:09.789912 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.789918 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.790103 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.790254 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:10:09.790618 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.791353 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.792410 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.792923 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.793029 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.793094 (+ 65us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.793180 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":97,"threads_started":1}
I20251025 20:10:09.794166 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.788847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:45135 (local address 127.0.0.1:39094)
1025 20:10:09.789654 (+ 807us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.789667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.789778 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.790899 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.790906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.790929 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:10:09.791212 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.791221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.792714 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.792718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.793738 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.793751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.793873 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.793892 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.793944 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.794009 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"spinlock_wait_cycles":3712,"thread_start_us":82,"threads_started":1}
I20251025 20:10:09.795408 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.789234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.18.82.65:45135)
1025 20:10:09.789598 (+ 364us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.789605 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.789809 (+ 204us) server_negotiation.cc:408] Connection header received
1025 20:10:09.790469 (+ 660us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.790475 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.790642 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.790763 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:10:09.791380 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.792568 (+ 1188us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.794357 (+ 1789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.794959 (+ 602us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.795075 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.795143 (+ 68us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.795236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20251025 20:10:09.796145 31128 tablet_service.cc:1505] Processing CreateTablet for tablet cd14fd243bac491a9a3333bfe73ad476 (DEFAULT_TABLE table=test-table [id=7969d3fa3941493b9efa75e8d071b61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:10:09.797036 31128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14fd243bac491a9a3333bfe73ad476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:10:09.800606 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.787652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:35255 (local address 127.0.0.1:37892)
1025 20:10:09.788550 (+ 898us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.788588 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.788721 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.798037 (+ 9316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.798041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.798060 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:10:09.798269 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.798281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.799575 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.799579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.800227 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.800238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.800376 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.800392 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.800439 (+ 47us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.800490 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":97,"threads_started":1}
I20251025 20:10:09.800242 30866 tablet_service.cc:1505] Processing CreateTablet for tablet cd14fd243bac491a9a3333bfe73ad476 (DEFAULT_TABLE table=test-table [id=7969d3fa3941493b9efa75e8d071b61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:10:09.801222 30866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14fd243bac491a9a3333bfe73ad476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:10:09.801649 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.787793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.18.82.66:35255)
1025 20:10:09.797451 (+ 9658us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.797458 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.797482 (+ 24us) server_negotiation.cc:408] Connection header received
1025 20:10:09.797593 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.797598 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.797763 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.797898 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:10:09.798416 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.799455 (+ 1039us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.800802 (+ 1347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.801275 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.801361 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.801413 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.801473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9544,"thread_start_us":107,"threads_started":1}
I20251025 20:10:09.803812 31226 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Bootstrap starting.
I20251025 20:10:09.804241 31006 tablet_service.cc:1505] Processing CreateTablet for tablet cd14fd243bac491a9a3333bfe73ad476 (DEFAULT_TABLE table=test-table [id=7969d3fa3941493b9efa75e8d071b61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 20:10:09.805075 31006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14fd243bac491a9a3333bfe73ad476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:10:09.807403 31226 tablet_bootstrap.cc:654] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Neither blocks nor log segments found. Creating new log.
I20251025 20:10:09.808682 31226 log.cc:826] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Log is configured to *not* fsync() on all Append() calls
I20251025 20:10:09.811084 31226 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: No bootstrap required, opened a new log
I20251025 20:10:09.811267 31226 ts_tablet_manager.cc:1403] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251025 20:10:09.811357 31228 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Bootstrap starting.
I20251025 20:10:09.813346 31228 tablet_bootstrap.cc:654] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Neither blocks nor log segments found. Creating new log.
I20251025 20:10:09.814489 31228 log.cc:826] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Log is configured to *not* fsync() on all Append() calls
I20251025 20:10:09.814471 31226 raft_consensus.cc:359] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.814682 31226 raft_consensus.cc:385] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:10:09.814739 31226 raft_consensus.cc:740] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397b718c2e9e4debae25e55083d685e7, State: Initialized, Role: FOLLOWER
I20251025 20:10:09.815120 31226 consensus_queue.cc:260] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.815732 31208 heartbeater.cc:499] Master 127.18.82.126:38381 was elected leader, sending a full tablet report...
I20251025 20:10:09.815896 31226 ts_tablet_manager.cc:1434] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 20:10:09.816329 31231 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: Bootstrap starting.
I20251025 20:10:09.818215 31231 tablet_bootstrap.cc:654] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: Neither blocks nor log segments found. Creating new log.
I20251025 20:10:09.818985 31231 log.cc:826] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: Log is configured to *not* fsync() on all Append() calls
I20251025 20:10:09.821235 31231 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: No bootstrap required, opened a new log
I20251025 20:10:09.821414 31231 ts_tablet_manager.cc:1403] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 20:10:09.823966 31228 tablet_bootstrap.cc:492] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: No bootstrap required, opened a new log
I20251025 20:10:09.824165 31228 ts_tablet_manager.cc:1403] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20251025 20:10:09.824668 31231 raft_consensus.cc:359] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.824882 31231 raft_consensus.cc:385] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:10:09.824982 31231 raft_consensus.cc:740] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d78f7a2f81a435c992d2cb98e10fada, State: Initialized, Role: FOLLOWER
I20251025 20:10:09.825443 31231 consensus_queue.cc:260] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.826395 31231 ts_tablet_manager.cc:1434] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 20:10:09.826777 30932 heartbeater.cc:499] Master 127.18.82.126:38381 was elected leader, sending a full tablet report...
I20251025 20:10:09.829008 31228 raft_consensus.cc:359] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.829166 31228 raft_consensus.cc:385] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:10:09.829228 31228 raft_consensus.cc:740] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d11b8a50f842708287978eaf950b85, State: Initialized, Role: FOLLOWER
I20251025 20:10:09.829707 31228 consensus_queue.cc:260] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.830291 31072 heartbeater.cc:499] Master 127.18.82.126:38381 was elected leader, sending a full tablet report...
I20251025 20:10:09.830476 31228 ts_tablet_manager.cc:1434] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20251025 20:10:09.876482 31073 tablet.cc:2378] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 20:10:09.893096 31233 raft_consensus.cc:493] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 20:10:09.893273 31233 raft_consensus.cc:515] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.894400 31233 leader_election.cc:290] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d11b8a50f842708287978eaf950b85 (127.18.82.66:35255), 397b718c2e9e4debae25e55083d685e7 (127.18.82.67:46227)
I20251025 20:10:09.897607 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.894558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.66:35255 (local address 127.18.82.65:41165)
1025 20:10:09.894886 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.894900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.895002 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.895243 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.895245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.895265 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:10:09.895573 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.895583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.896364 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.896367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.897142 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.897150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.897259 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.897280 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.897323 (+ 43us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.897370 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20251025 20:10:09.898083 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.894888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:41165 (local address 127.18.82.66:35255)
1025 20:10:09.895016 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.895020 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.895034 (+ 14us) server_negotiation.cc:408] Connection header received
1025 20:10:09.895084 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.895088 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.895125 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.895188 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:10:09.895703 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.896250 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.897272 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.897813 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.897857 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.897907 (+ 50us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.897964 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 20:10:09.898440 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.894771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:60401 (local address 127.18.82.67:46227)
1025 20:10:09.894988 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.894993 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.895191 (+ 198us) server_negotiation.cc:408] Connection header received
1025 20:10:09.895393 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.895398 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.895448 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.895520 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 20:10:09.896005 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.896557 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.897574 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.898060 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.898090 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.898290 (+ 200us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.898358 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20251025 20:10:09.898752 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.894655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.67:46227 (local address 127.18.82.65:60401)
1025 20:10:09.895101 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.895113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:09.895209 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:09.895558 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:09.895561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:09.895582 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 20:10:09.895881 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.895889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.896672 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.896675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:09.897423 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:09.897431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.898139 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.898157 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 20:10:09.898604 (+ 447us) client_negotiation.cc:240] Negotiation successful
1025 20:10:09.898664 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":95,"threads_started":1}
I20251025 20:10:09.898772 31014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd14fd243bac491a9a3333bfe73ad476" candidate_uuid: "9d78f7a2f81a435c992d2cb98e10fada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d11b8a50f842708287978eaf950b85" is_pre_election: true
I20251025 20:10:09.899044 31014 raft_consensus.cc:2468] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d78f7a2f81a435c992d2cb98e10fada in term 0.
I20251025 20:10:09.899399 31161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd14fd243bac491a9a3333bfe73ad476" candidate_uuid: "9d78f7a2f81a435c992d2cb98e10fada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397b718c2e9e4debae25e55083d685e7" is_pre_election: true
I20251025 20:10:09.899533 30820 leader_election.cc:304] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d78f7a2f81a435c992d2cb98e10fada, c1d11b8a50f842708287978eaf950b85; no voters:
I20251025 20:10:09.899636 31161 raft_consensus.cc:2468] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d78f7a2f81a435c992d2cb98e10fada in term 0.
I20251025 20:10:09.899791 31233 raft_consensus.cc:2804] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 20:10:09.899895 31233 raft_consensus.cc:493] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 20:10:09.899957 31233 raft_consensus.cc:3060] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:10:09.900879 31233 raft_consensus.cc:515] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.901237 31233 leader_election.cc:290] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [CANDIDATE]: Term 1 election: Requested vote from peers c1d11b8a50f842708287978eaf950b85 (127.18.82.66:35255), 397b718c2e9e4debae25e55083d685e7 (127.18.82.67:46227)
I20251025 20:10:09.901607 31014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd14fd243bac491a9a3333bfe73ad476" candidate_uuid: "9d78f7a2f81a435c992d2cb98e10fada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d11b8a50f842708287978eaf950b85"
I20251025 20:10:09.901607 31161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd14fd243bac491a9a3333bfe73ad476" candidate_uuid: "9d78f7a2f81a435c992d2cb98e10fada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397b718c2e9e4debae25e55083d685e7"
I20251025 20:10:09.901723 31161 raft_consensus.cc:3060] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:10:09.901723 31014 raft_consensus.cc:3060] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:10:09.902827 31161 raft_consensus.cc:2468] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d78f7a2f81a435c992d2cb98e10fada in term 1.
I20251025 20:10:09.902827 31014 raft_consensus.cc:2468] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d78f7a2f81a435c992d2cb98e10fada in term 1.
I20251025 20:10:09.903119 30820 leader_election.cc:304] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d78f7a2f81a435c992d2cb98e10fada, c1d11b8a50f842708287978eaf950b85; no voters:
I20251025 20:10:09.903316 31233 raft_consensus.cc:2804] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:10:09.903507 31233 raft_consensus.cc:697] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [term 1 LEADER]: Becoming Leader. State: Replica: 9d78f7a2f81a435c992d2cb98e10fada, State: Running, Role: LEADER
I20251025 20:10:09.903801 31233 consensus_queue.cc:237] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } }
I20251025 20:10:09.906885 30719 catalog_manager.cc:5649] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d78f7a2f81a435c992d2cb98e10fada (127.18.82.65). New cstate: current_term: 1 leader_uuid: "9d78f7a2f81a435c992d2cb98e10fada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d78f7a2f81a435c992d2cb98e10fada" member_type: VOTER last_known_addr { host: "127.18.82.65" port: 45135 } health_report { overall_health: HEALTHY } } }
I20251025 20:10:09.926853 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.923011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.18.82.65:45135)
1025 20:10:09.923121 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:09.923124 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:09.923137 (+ 13us) server_negotiation.cc:408] Connection header received
1025 20:10:09.923216 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:09.923223 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:09.923286 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:09.923375 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 20:10:09.923810 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.924552 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:09.925350 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:09.925565 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:09.925667 (+ 102us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 20:10:09.926179 (+ 512us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 20:10:09.926320 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:09.926677 (+ 357us) server_negotiation.cc:300] Negotiation successful
1025 20:10:09.926733 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251025 20:10:09.934693 31161 raft_consensus.cc:1275] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 1 FOLLOWER]: Refusing update from remote peer 9d78f7a2f81a435c992d2cb98e10fada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:10:09.934693 31014 raft_consensus.cc:1275] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 1 FOLLOWER]: Refusing update from remote peer 9d78f7a2f81a435c992d2cb98e10fada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 20:10:09.935364 31237 consensus_queue.cc:1048] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [LEADER]: Connected to new peer: Peer: permanent_uuid: "397b718c2e9e4debae25e55083d685e7" member_type: VOTER last_known_addr { host: "127.18.82.67" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:10:09.935696 31237 consensus_queue.cc:1048] T cd14fd243bac491a9a3333bfe73ad476 P 9d78f7a2f81a435c992d2cb98e10fada [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d11b8a50f842708287978eaf950b85" member_type: VOTER last_known_addr { host: "127.18.82.66" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 20:10:09.942202 31240 mvcc.cc:204] Tried to move back new op lower bound from 7214788648685715456 to 7214788648568410112. Current Snapshot: MvccSnapshot[applied={T|T < 7214788648685715456}]
I20251025 20:10:09.945034 31242 mvcc.cc:204] Tried to move back new op lower bound from 7214788648685715456 to 7214788648568410112. Current Snapshot: MvccSnapshot[applied={T|T < 7214788648685715456}]
I20251025 20:10:09.945454 31241 mvcc.cc:204] Tried to move back new op lower bound from 7214788648685715456 to 7214788648568410112. Current Snapshot: MvccSnapshot[applied={T|T < 7214788648685715456}]
I20251025 20:10:09.954118 30718 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39678:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:10:09.954265 30718 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39678:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 20:10:09.956538 30718 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b06bea56e66246c0b5e9ccbcdacb2a8b: Sending DeleteTablet for 3 replicas of tablet cd14fd243bac491a9a3333bfe73ad476
I20251025 20:10:09.957352 30866 tablet_service.cc:1552] Processing DeleteTablet for tablet cd14fd243bac491a9a3333bfe73ad476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:10:09 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:39094
I20251025 20:10:09.957352 31128 tablet_service.cc:1552] Processing DeleteTablet for tablet cd14fd243bac491a9a3333bfe73ad476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:10:09 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:40734
I20251025 20:10:09.957368 31006 tablet_service.cc:1552] Processing DeleteTablet for tablet cd14fd243bac491a9a3333bfe73ad476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 20:10:09 UTC) from {username='kudu', principal='kudu/127.18.82.126@KRBTEST.COM'} at 127.0.0.1:37892
I20251025 20:10:09.958225 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 30801
I20251025 20:10:09.959635 31252 tablet_replica.cc:333] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: stopping tablet replica
I20251025 20:10:09.959846 31252 raft_consensus.cc:2243] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:10:09.960091 31252 raft_consensus.cc:2272] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 20:10:09.961198 31252 ts_tablet_manager.cc:1916] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:10:09.963189 31252 ts_tablet_manager.cc:1929] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:10:09.963332 31252 log.cc:1199] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-1/wal/wals/cd14fd243bac491a9a3333bfe73ad476
I20251025 20:10:09.963703 31252 ts_tablet_manager.cc:1950] T cd14fd243bac491a9a3333bfe73ad476 P c1d11b8a50f842708287978eaf950b85: Deleting consensus metadata
I20251025 20:10:09.964820 30706 catalog_manager.cc:4985] TS c1d11b8a50f842708287978eaf950b85 (127.18.82.66:35255): tablet cd14fd243bac491a9a3333bfe73ad476 (table test-table [id=7969d3fa3941493b9efa75e8d071b61a]) successfully deleted
I20251025 20:10:09.968367 31251 tablet_replica.cc:333] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: stopping tablet replica
I20251025 20:10:09.968606 31251 raft_consensus.cc:2243] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 20:10:09.968909 31251 raft_consensus.cc:2272] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 20:10:09.968967 30706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.82.65:45135 (error 108)
I20251025 20:10:09.969795 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 30940
I20251025 20:10:09.971035 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:09.970672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.65:45135 (local address 127.0.0.1:39108)
1025 20:10:09.970875 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:09.970936 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.65:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20251025 20:10:09.971176 30706 catalog_manager.cc:4712] TS 9d78f7a2f81a435c992d2cb98e10fada (127.18.82.65:45135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd14fd243bac491a9a3333bfe73ad476: Network error: Client connection negotiation failed: client connection to 127.18.82.65:45135: connect: Connection refused (error 111)
I20251025 20:10:09.972602 31251 ts_tablet_manager.cc:1916] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 20:10:09.975575 31251 ts_tablet_manager.cc:1929] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 20:10:09.975668 31251 log.cc:1199] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Deleting WAL directory at /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761422869575105-18761-0/minicluster-data/ts-2/wal/wals/cd14fd243bac491a9a3333bfe73ad476
I20251025 20:10:09.976014 31251 ts_tablet_manager.cc:1950] T cd14fd243bac491a9a3333bfe73ad476 P 397b718c2e9e4debae25e55083d685e7: Deleting consensus metadata
I20251025 20:10:09.977161 30703 catalog_manager.cc:4985] TS 397b718c2e9e4debae25e55083d685e7 (127.18.82.67:46227): tablet cd14fd243bac491a9a3333bfe73ad476 (table test-table [id=7969d3fa3941493b9efa75e8d071b61a]) successfully deleted
I20251025 20:10:09.978767 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 31079
I20251025 20:10:09.984839 18761 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskKRacCB/build/debug/bin/kudu with pid 30687
I20251025 20:10:09.991621 18761 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251025 20:10:10.337267 18761 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251025 20:10:10.337397 18761 mini_ranger.cc:66] Stopping Ranger...
I20251025 20:10:10.683002 18761 mini_ranger.cc:68] Stopped Ranger
2025-10-25 20:10:10.683 UTC [30247] LOG: received smart shutdown request
2025-10-25 20:10:10.685 UTC [30247] LOG: background worker "logical replication launcher" (PID 30255) exited with exit code 1
2025-10-25 20:10:10.686 UTC [30250] LOG: shutting down
2025-10-25 20:10:10.688 UTC [30250] LOG: checkpoint starting: shutdown immediate
2025-10-25 20:10:11.480 UTC [30250] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.035 s, sync=0.752 s, total=0.794 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/2044D28, redo lsn=0/2044D28
2025-10-25 20:10:11.488 UTC [30247] LOG: database system is shut down
2025-10-25T20:10:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46668 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 20:10:11 dist-test-slave-d2pk krb5kdc[31262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 20:10:11 dist-test-slave-d2pk krb5kdc[31262](info): set up 2 sockets
Oct 25 20:10:11 dist-test-slave-d2pk krb5kdc[31262](info): commencing operation
krb5kdc: starting...
W20251025 20:10:13.421748 18761 mini_kdc.cc:121] Time spent starting KDC: real 1.802s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/test-user.keytab.
Oct 25 20:10:13 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T20:10:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T20:10:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangeradmin_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangeradmin_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangerlookup_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangerlookup_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
I20251025 20:10:13.559352 18761 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKRacCB/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/postgres -l logfile start
2025-10-25 20:10:15.764 UTC [31293] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-25 20:10:15.764 UTC [31293] LOG: listening on IPv4 address "127.18.82.84", port 36411
2025-10-25 20:10:15.767 UTC [31293] LOG: listening on Unix socket "/tmp/.s.PGSQL.36411"
2025-10-25 20:10:15.773 UTC [31298] LOG: database system was shut down at 2025-10-25 20:10:14 UTC
2025-10-25 20:10:15.778 UTC [31293] LOG: database system is ready to accept connections
I20251025 20:10:17.517536 18761 mini_postgres.cc:96] Postgres bound to 36411
2025-10-25 20:10:17.523 UTC [31304] FATAL: database "slave" does not exist
127.18.82.84:36411 - accepting connections
I20251025 20:10:17.524205 18761 mini_ranger.cc:161] Starting Ranger...
I20251025 20:10:17.540522 18761 mini_ranger.cc:84] Created miniranger Postgres user
I20251025 20:10:17.602435 18761 mini_ranger.cc:87] Created ranger Postgres database
I20251025 20:10:17.602541 18761 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin
2025-10-25 20:10:17,879 [I] DB FLAVOR :POSTGRES
2025-10-25 20:10:17,880 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 20:10:17,880 [I] Checking connection..
2025-10-25 20:10:17,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-25 20:10:18,199 [I] Checking connection passed.
2025-10-25 20:10:18,199 [I] --------- Verifying version history table ---------
2025-10-25 20:10:18,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:10:18,536 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 20:10:18,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:10:18,844 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 20:10:18,845 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-25 20:10:18,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-25 20:10:19.145 UTC [31371] WARNING: there is no transaction in progress
2025-10-25 20:10:19,151 [I] create_dbversion_catalog.sql file imported successfully
2025-10-25 20:10:19,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 20:10:19,437 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-25 20:10:19,438 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 20:10:19,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-25 20:10:19,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-25 20:10:20,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d2pk.c.gcp-upstream.internal','N') ;"
2025-10-25 20:10:20,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 20:10:20,634 [I] Table x_portal_user does not exist in database ranger
2025-10-25 20:10:20,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 20:10:20,920 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-25 20:10:20,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 20:10:21,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 20:10:21,519 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-25 20:10:21,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-25 20:10:22.273 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.282 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.292 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.301 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.311 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.320 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.360 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.364 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.373 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.383 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.391 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.399 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.406 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.410 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.413 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22.416 UTC [31488] WARNING: there is no transaction in progress
2025-10-25 20:10:22,774 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-25 20:10:22,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-25 20:10:23,067 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-25 20:10:23,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 20:10:23,367 [I] Table x_portal_user already exists in database 'ranger'
2025-10-25 20:10:23,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 20:10:23,667 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-25 20:10:23,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 20:10:23,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 20:10:24,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d2pk.c.gcp-upstream.internal';"
2025-10-25 20:10:24,545 [I] CORE_DB_SCHEMA import status has been updated
2025-10-25 20:10:24,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-25 20:10:24,850 [I] DB_PATCHES have already been applied
I20251025 20:10:24.856264 18761 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKRacCB/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251025 20:10:24.856335 18761 mini_ranger.cc:193] Using host: 127.18.82.84
I20251025 20:10:24.859707 18761 mini_ranger.cc:227] Ranger admin URL: http://127.18.82.84:44447
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 25, 2025 8:10:25 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-admin/ews/webapp].
Oct 25, 2025 8:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 25, 2025 8:10:25 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-44447"]
Oct 25, 2025 8:10:25 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:10:25 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:10:25 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 25, 2025 8:10:25 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 25, 2025 8:10:25 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 25, 2025 8:10:26 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 25, 2025 8:10:26 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25, 2025 8:10:26 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 25, 2025 8:10:26 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251025 20:10:26.920357 18761 mini_ranger.cc:160] Time spent starting Ranger: real 9.396s user 0.001s sys 0.009s
Oct 25, 2025 8:10:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 25, 2025 8:10:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 25, 2025 8:10:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 25, 2025 8:10:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 25, 2025 8:10:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 25, 2025 8:10:45 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-44447"]
I20251025 20:10:45.681260 18761 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.82.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.82.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangerkms_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.82.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/rangerkms_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.82.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.82.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.82.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/HTTP_127.18.82.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 25 20:10:45 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251025 20:10:45.756695 18761 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251025 20:10:45.765272 18761 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251025 20:10:45.870955 18761 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251025 20:10:45.871049 18761 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms
I20251025 20:10:45.871069 18761 mini_ranger_kms.cc:227] Using postgres at 127.18.82.84:36411
2025-10-25 20:10:46,061 [I] DB FLAVOR :POSTGRES
2025-10-25 20:10:46,061 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 20:10:46,061 [I] Checking connection
2025-10-25 20:10:46,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-25 20:10:46,413 [I] connection success
2025-10-25 20:10:46,413 [I] --------- Verifying Ranger DB tables ---------
2025-10-25 20:10:46,413 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-25 20:10:46,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-25 20:10:46,714 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-25 20:10:46,715 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 20:10:46,715 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-25 20:10:46,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.82.84:36411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-25 20:10:47,016 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251025 20:10:47.479174 18761 mini_ranger_kms.cc:310] Created kms service
I20251025 20:10:47.677284 18761 mini_ranger_kms.cc:326] Created kudu user
I20251025 20:10:47.716879 18761 mini_ranger_kms.cc:343] Created rangerkms user
Oct 25 20:10:47 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:48.041715 18761 mini_ranger_kms.cc:384] Added ranger policy
I20251025 20:10:48.042040 18761 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKRacCB/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKRacCB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251025 20:10:48.042068 18761 mini_ranger_kms.cc:240] Using host: 127.18.82.84
I20251025 20:10:48.045773 18761 mini_ranger_kms.cc:276] Ranger KMS PID: 31680
I20251025 20:10:48.045850 18761 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.82.84:44141
Oct 25, 2025 8:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:10:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:10:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 25, 2025 8:10:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 25, 2025 8:10:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-kms/ews/webapp].
Oct 25, 2025 8:10:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 25, 2025 8:10:49 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-44141"]
Oct 25, 2025 8:10:49 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:10:49 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:10:49 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 25, 2025 8:10:49 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25 20:10:49 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761423049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:50.114116 18761 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.357s user 0.002s sys 0.172s
I20251025 20:10:50.114243 18761 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251025 20:10:50.114312 18761 mini_ranger_kms.cc:401] 127.18.82.84:44141/kms/v1/keys
Oct 25 20:10:50 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
Oct 25 20:10:50 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.82.84@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
Oct 25 20:10:52 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:52.853482 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:41523
--webserver_interface=127.18.82.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.82.126:41523
--ranger_config_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:44141/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:52.979437 31733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:52.979655 31733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:52.979709 31733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:52.983039 31733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 20:10:52.983100 31733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:52.983124 31733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:52.983144 31733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 20:10:52.983189 31733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 20:10:52.987596 31733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.82.84:44141/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.82.126:41523
--ranger_config_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.82.126:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.82.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:52.988765 31733 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:52.989683 31733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:52.995553 31739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:52.995715 31733 server_base.cc:1047] running on GCE node
W20251025 20:10:52.995582 31738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:52.995545 31741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:52.996439 31733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:52.997339 31733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:52.998492 31733 hybrid_clock.cc:648] HybridClock initialized: now 1761423052998485 us; error 50 us; skew 500 ppm
Oct 25 20:10:52 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423052, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:53.001389 31733 init.cc:377] Logged in from keytab as kudu/127.18.82.126@KRBTEST.COM (short username kudu)
I20251025 20:10:53.002563 31733 webserver.cc:492] Webserver started at http://127.18.82.126:36451/ using document root <none> and password file <none>
I20251025 20:10:53.003093 31733 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:53.003147 31733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:53.003333 31733 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:52 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423052, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.126@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:53.045926 31733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data/instance:
uuid: "d9b1f2e3d52c45acbb04db28e12243f3"
format_stamp: "Formatted at 2025-10-25 20:10:53 on dist-test-slave-d2pk"
server_key: "f38bdc148781a4dfdb7f8fafa3cfceab"
server_key_iv: "7dc985d330c7b9c2da7311ddd9831fe8"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:53.046525 31733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal/instance:
uuid: "d9b1f2e3d52c45acbb04db28e12243f3"
format_stamp: "Formatted at 2025-10-25 20:10:53 on dist-test-slave-d2pk"
server_key: "f38bdc148781a4dfdb7f8fafa3cfceab"
server_key_iv: "7dc985d330c7b9c2da7311ddd9831fe8"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:53.050346 31733 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251025 20:10:53.063469 31748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:53.064692 31733 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 20:10:53.064855 31733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal
uuid: "d9b1f2e3d52c45acbb04db28e12243f3"
format_stamp: "Formatted at 2025-10-25 20:10:53 on dist-test-slave-d2pk"
server_key: "f38bdc148781a4dfdb7f8fafa3cfceab"
server_key_iv: "7dc985d330c7b9c2da7311ddd9831fe8"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:53.064968 31733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:53.085759 31733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:53.088668 31733 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:53.088847 31733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:53.097342 31733 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.126:41523
I20251025 20:10:53.097352 31800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.126:41523 every 8 connection(s)
I20251025 20:10:53.098335 31733 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/data/info.pb
I20251025 20:10:53.100054 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 31733
I20251025 20:10:53.100167 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/wal/instance
I20251025 20:10:53.100446 18761 external_mini_cluster.cc:1468] Setting key d9a1f63eadab8ef5f155a58589e5e481
I20251025 20:10:53.102401 31801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 20:10:53.109311 31801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3: Bootstrap starting.
Oct 25 20:10:53 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
I20251025 20:10:53.111949 31801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3: Neither blocks nor log segments found. Creating new log.
I20251025 20:10:53.112747 31801 log.cc:826] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3: Log is configured to *not* fsync() on all Append() calls
I20251025 20:10:53.114802 31801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3: No bootstrap required, opened a new log
I20251025 20:10:53.115345 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:53.101914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.18.82.126:41523)
1025 20:10:53.102294 (+ 380us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:53.102305 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:53.102343 (+ 38us) server_negotiation.cc:408] Connection header received
1025 20:10:53.103226 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:53.103258 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:53.103665 (+ 407us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:53.104035 (+ 370us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:53.105104 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:53.106365 (+ 1261us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:53.107347 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:53.107620 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:53.110252 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:53.110300 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:53.110314 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:53.110350 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:53.112229 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:53.112653 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:53.112658 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:53.112663 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:53.112734 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:53.112979 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:53.112983 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:53.112985 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:53.113370 (+ 385us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:53.113672 (+ 302us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:53.113954 (+ 282us) server_negotiation.cc:300] Negotiation successful
1025 20:10:53.114227 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20251025 20:10:53.118584 31801 raft_consensus.cc:359] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } }
I20251025 20:10:53.118767 31801 raft_consensus.cc:385] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 20:10:53.118831 31801 raft_consensus.cc:740] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b1f2e3d52c45acbb04db28e12243f3, State: Initialized, Role: FOLLOWER
I20251025 20:10:53.119262 31801 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } }
I20251025 20:10:53.119387 31801 raft_consensus.cc:399] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 20:10:53.119465 31801 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 20:10:53.119551 31801 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 0 FOLLOWER]: Advancing to term 1
I20251025 20:10:53.120332 31801 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } }
I20251025 20:10:53.120645 31801 leader_election.cc:304] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b1f2e3d52c45acbb04db28e12243f3; no voters:
I20251025 20:10:53.120968 31801 leader_election.cc:290] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 20:10:53.121101 31806 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 20:10:53.121351 31806 raft_consensus.cc:697] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [term 1 LEADER]: Becoming Leader. State: Replica: d9b1f2e3d52c45acbb04db28e12243f3, State: Running, Role: LEADER
I20251025 20:10:53.121726 31806 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } }
I20251025 20:10:53.122335 31801 sys_catalog.cc:565] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 20:10:53.122942 31806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } } }
I20251025 20:10:53.123106 31806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [sys.catalog]: This master's current role is: LEADER
I20251025 20:10:53.123322 31806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b1f2e3d52c45acbb04db28e12243f3. Latest consensus state: current_term: 1 leader_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b1f2e3d52c45acbb04db28e12243f3" member_type: VOTER last_known_addr { host: "127.18.82.126" port: 41523 } } }
I20251025 20:10:53.123450 31806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3 [sys.catalog]: This master's current role is: LEADER
I20251025 20:10:53.123730 31813 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 20:10:53.127305 31813 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 20:10:53.127732 31801 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251025 20:10:53.132926 31813 catalog_manager.cc:1357] Generated new cluster ID: 48032c27b4f645b5880487efd6e12473
I20251025 20:10:53.133034 31813 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 20:10:53.151835 31813 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 20:10:53.153252 31813 catalog_manager.cc:1514] Loading token signing keys...
I20251025 20:10:53.162743 31813 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d9b1f2e3d52c45acbb04db28e12243f3: Generated new TSK 0
I20251025 20:10:53.163841 31813 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:53.871927 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:44141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.65:0
--local_ip_for_outbound_sockets=127.18.82.65
--webserver_interface=127.18.82.65
--webserver_port=0
--tserver_master_addrs=127.18.82.126:41523
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 20:10:53.980163 31847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:53.980351 31847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:53.980399 31847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:53.983561 31847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:53.983628 31847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:53.983728 31847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.65
I20251025 20:10:53.987926 31847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.82.84:44141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.82.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:41523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.65
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:53.988936 31847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:53.990023 31847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:53.996898 31854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:53.996929 31855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:53.997020 31847 server_base.cc:1047] running on GCE node
W20251025 20:10:53.997144 31857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:53.997614 31847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:53.998101 31847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:53.999254 31847 hybrid_clock.cc:648] HybridClock initialized: now 1761423053999240 us; error 31 us; skew 500 ppm
Oct 25 20:10:53 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423053, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:54.001767 31847 init.cc:377] Logged in from keytab as kudu/127.18.82.65@KRBTEST.COM (short username kudu)
I20251025 20:10:54.002748 31847 webserver.cc:492] Webserver started at http://127.18.82.65:38013/ using document root <none> and password file <none>
I20251025 20:10:54.003259 31847 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:54.003306 31847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:54.003472 31847 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:53 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423053, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:54.017144 31847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data/instance:
uuid: "ea98acb75708486d9d6c42b432874b1f"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "a9ee94ae853688e4f5440069562f9cab"
server_key_iv: "0ccacfad94241579078eb74e83f39a4d"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.017799 31847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance:
uuid: "ea98acb75708486d9d6c42b432874b1f"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "a9ee94ae853688e4f5440069562f9cab"
server_key_iv: "0ccacfad94241579078eb74e83f39a4d"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.022403 31847 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20251025 20:10:54.057884 31864 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.059291 31847 fs_manager.cc:730] Time spent opening block manager: real 0.022s user 0.001s sys 0.001s
I20251025 20:10:54.059417 31847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal
uuid: "ea98acb75708486d9d6c42b432874b1f"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "a9ee94ae853688e4f5440069562f9cab"
server_key_iv: "0ccacfad94241579078eb74e83f39a4d"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.059537 31847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:54.076692 31847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:54.079938 31847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:54.080093 31847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:54.080610 31847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:10:54.081447 31847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:10:54.081501 31847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.081589 31847 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:10:54.081636 31847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.091621 31847 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.65:35285
I20251025 20:10:54.092757 31847 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/data/info.pb
I20251025 20:10:54.094655 31979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.65:35285 every 8 connection(s)
I20251025 20:10:54.100467 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 31847
I20251025 20:10:54.100632 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-0/wal/instance
I20251025 20:10:54.100894 18761 external_mini_cluster.cc:1468] Setting key 83c4be84af1ca2cedf6e2a437c05b681
WARNING: no policy specified for kudu/127.18.82.66@KRBTEST.COM; defaulting to no policy
Oct 25 20:10:54 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423053, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.65@KRBTEST.COM for kudu/127.18.82.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.82.66@KRBTEST.COM" created.
I20251025 20:10:54.122961 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:54.099079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.82.65:51939 (local address 127.18.82.126:41523)
1025 20:10:54.099382 (+ 303us) server_negotiation.cc:207] Beginning negotiation
1025 20:10:54.099387 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 20:10:54.105926 (+ 6539us) server_negotiation.cc:408] Connection header received
1025 20:10:54.108577 (+ 2651us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 20:10:54.108581 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 20:10:54.108642 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 20:10:54.108707 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 20:10:54.110324 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:54.110811 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 20:10:54.111360 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 20:10:54.111549 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:54.118753 (+ 7204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 20:10:54.118777 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 20:10:54.118781 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 20:10:54.118811 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 20:10:54.120678 (+ 1867us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:54.121232 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:54.121237 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:54.121239 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:54.121294 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 20:10:54.121695 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 20:10:54.121698 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 20:10:54.121700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 20:10:54.121875 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 20:10:54.121951 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1025 20:10:54.122652 (+ 701us) server_negotiation.cc:300] Negotiation successful
1025 20:10:54.122784 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20251025 20:10:54.127895 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:54.098105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:51939)
1025 20:10:54.105800 (+ 7695us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:54.105839 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:54.106773 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:54.109137 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 20:10:54.109148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 20:10:54.109542 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1025 20:10:54.110150 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:54.110162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:54.110920 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 20:10:54.110923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 20:10:54.111259 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 20:10:54.111264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 20:10:54.111473 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 20:10:54.115156 (+ 3683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 20:10:54.115200 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1025 20:10:54.118564 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 20:10:54.120810 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:54.120816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:54.120828 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:54.121108 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:54.121404 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 20:10:54.121407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 20:10:54.121409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 20:10:54.121574 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 20:10:54.122013 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 20:10:54.122020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 20:10:54.122317 (+ 297us) client_negotiation.cc:769] Sending connection context
1025 20:10:54.123415 (+ 1098us) client_negotiation.cc:240] Negotiation successful
1025 20:10:54.126968 (+ 3553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7481,"thread_start_us":111,"threads_started":1}
I20251025 20:10:54.129319 31980 heartbeater.cc:344] Connected to a master server at 127.18.82.126:41523
I20251025 20:10:54.129603 31980 heartbeater.cc:461] Registering TS with master...
I20251025 20:10:54.130255 31980 heartbeater.cc:507] Master 127.18.82.126:41523 requested a full tablet report, sending...
I20251025 20:10:54.132248 31765 ts_manager.cc:194] Registered new tserver with Master: ea98acb75708486d9d6c42b432874b1f (127.18.82.65:35285)
I20251025 20:10:54.133841 31765 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.82.65@KRBTEST.COM'} at 127.18.82.65:51939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.82.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.82.66@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.18.82.126 from keytab /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.82.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab.
I20251025 20:10:54.161908 18761 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskKRacCB/build/debug/bin/kudu
/tmp/dist-test-taskKRacCB/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.82.84:44141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.82.66:0
--local_ip_for_outbound_sockets=127.18.82.66
--webserver_interface=127.18.82.66
--webserver_port=0
--tserver_master_addrs=127.18.82.126:41523
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.82.84:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 20:10:54.179371 31839 server.cc:273] Received an EOF from the subprocess
F20251025 20:10:54.180284 31826 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f675237cdcd google::LogMessage::Fail() at ??:0
@ 0x7f6752380b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f675237c7cc google::LogMessage::Flush() at ??:0
@ 0x7f675237df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f67540322d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f675402f0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f6754033005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557515c98bea std::function<>::operator()() at ??:0
@ 0x7f67537d6598 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f675406e6db start_thread at ??:0
@ 0x7f675153371f clone at ??:0
W20251025 20:10:54.273938 31989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 20:10:54.274142 31989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 20:10:54.274230 31989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 20:10:54.277271 31989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 20:10:54.277323 31989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 20:10:54.277410 31989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.82.66
I20251025 20:10:54.281098 31989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.82.84:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.82.84:44141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.82.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.82.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.82.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.82.126:41523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.82.66
--log_dir=/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 19:43:16 UTC on 24a791456cd2
build id 8714
I20251025 20:10:54.282030 31989 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 20:10:54.282828 31989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 20:10:54.288842 32006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:54.288856 32005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 20:10:54.288878 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 20:10:54.289172 31989 server_base.cc:1047] running on GCE node
I20251025 20:10:54.289479 31989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 20:10:54.289981 31989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 20:10:54.291121 31989 hybrid_clock.cc:648] HybridClock initialized: now 1761423054291100 us; error 38 us; skew 500 ppm
Oct 25 20:10:54 dist-test-slave-d2pk krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423054, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 20:10:54.293826 31989 init.cc:377] Logged in from keytab as kudu/127.18.82.66@KRBTEST.COM (short username kudu)
I20251025 20:10:54.294728 31989 webserver.cc:492] Webserver started at http://127.18.82.66:39283/ using document root <none> and password file <none>
I20251025 20:10:54.295204 31989 fs_manager.cc:362] Metadata directory not provided
I20251025 20:10:54.295282 31989 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 20:10:54.295454 31989 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 20:10:54 dist-test-slave-d2pk krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761423054, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.82.66@KRBTEST.COM for HTTP/127.18.82.84@KRBTEST.COM
I20251025 20:10:54.308277 31989 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data/instance:
uuid: "986d450dca7940fb9fc52010cba87bce"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "bd0b5dbe9c9b429779ff858e34866e34"
server_key_iv: "26d5d262eaced975f3dcf2c0f9d9d83a"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.309069 31989 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance:
uuid: "986d450dca7940fb9fc52010cba87bce"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "bd0b5dbe9c9b429779ff858e34866e34"
server_key_iv: "26d5d262eaced975f3dcf2c0f9d9d83a"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.313231 31989 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251025 20:10:54.327126 32015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.328446 31989 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251025 20:10:54.328630 31989 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal
uuid: "986d450dca7940fb9fc52010cba87bce"
format_stamp: "Formatted at 2025-10-25 20:10:54 on dist-test-slave-d2pk"
server_key: "bd0b5dbe9c9b429779ff858e34866e34"
server_key_iv: "26d5d262eaced975f3dcf2c0f9d9d83a"
server_key_version: "kuduclusterkey@0"
I20251025 20:10:54.328804 31989 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 20:10:54.363510 31989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 20:10:54.366514 31989 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 20:10:54.366680 31989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 20:10:54.367228 31989 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 20:10:54.368150 31989 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 20:10:54.368196 31989 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.368249 31989 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 20:10:54.368278 31989 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 20:10:54.378543 31989 rpc_server.cc:307] RPC server started. Bound to: 127.18.82.66:46587
I20251025 20:10:54.378633 32128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.82.66:46587 every 8 connection(s)
I20251025 20:10:54.379410 31989 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/data/info.pb
I20251025 20:10:54.387981 18761 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskKRacCB/build/debug/bin/kudu as pid 31989
I20251025 20:10:54.388095 18761 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskKRacCB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761422869575105-18761-0/minicluster-data/ts-1/wal/instance
I20251025 20:10:54.388336 18761 external_mini_cluster.cc:1468] Setting key 97217794b6b168bd53d5afa41eac441e
W20251025 20:10:54.432082 31260 connection.cc:537] client connection to 127.18.82.126:41523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 20:10:54.433176 32131 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 20:10:54.381246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:58919)
1025 20:10:54.382056 (+ 810us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:54.382095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 20:10:54.382768 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 20:10:54.432467 (+ 49699us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":513,"threads_started":1}
W20251025 20:10:54.433702 32129 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:41523 (0 consecutive failures): Network error: Failed to ping master at 127.18.82.126:41523: Client connection negotiation failed: client connection to 127.18.82.126:41523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 20:10:54.434458 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:54.434112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:60809)
1025 20:10:54.434294 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:54.434364 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20251025 20:10:54.435096 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:54.434880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:48443)
1025 20:10:54.434988 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:54.435036 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251025 20:10:54.435251 32129 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251025 20:10:55.136657 31980 heartbeater.cc:499] Master 127.18.82.126:41523 was elected leader, sending a full tablet report...
I20251025 20:10:55.137627 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:55.137162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:44643)
1025 20:10:55.137421 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:55.137486 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
W20251025 20:10:55.137877 31980 heartbeater.cc:646] Failed to heartbeat to 127.18.82.126:41523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
I20251025 20:10:55.436360 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:55.435891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:45899)
1025 20:10:55.436173 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:55.436234 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20251025 20:10:56.138871 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:56.138432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:54655)
1025 20:10:56.138712 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:56.138762 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20251025 20:10:56.437492 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:56.437056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:49667)
1025 20:10:56.437296 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:56.437339 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20251025 20:10:57.140197 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:57.139784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:48547)
1025 20:10:57.140048 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:57.140106 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20251025 20:10:57.438670 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:57.438223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:45999)
1025 20:10:57.438498 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:57.438552 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20251025 20:10:58.141711 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:58.141074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:39959)
1025 20:10:58.141494 (+ 420us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:58.141594 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":99,"threads_started":1}
I20251025 20:10:58.439839 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:58.439385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:52825)
1025 20:10:58.439684 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:58.439740 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20251025 20:10:59.143237 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:59.142651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.65:56401)
1025 20:10:59.143061 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:59.143121 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":85,"threads_started":1}
I20251025 20:10:59.441322 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 20:10:59.440863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.82.126:41523 (local address 127.18.82.66:48099)
1025 20:10:59.441157 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1025 20:10:59.441226 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.82.126:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
F20251025 20:10:59.486884 18761 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1761423059 (unix time) try "date -d @1761423059" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004949) received by PID 18761 (TID 0x7fdf13573d80) from PID 18761; stack trace: ***
@ 0x7fdf1f20d980 (unknown) at ??:0
@ 0x7fdf1c845fb7 gsignal at ??:0
@ 0x7fdf1c847921 abort at ??:0
@ 0x7fdf1dab3dcd google::LogMessage::Fail() at ??:0
@ 0x7fdf1dab7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdf1dab37cc google::LogMessage::Flush() at ??:0
@ 0x7fdf1dab4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdf2083bcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fdf208389c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55e908a0792f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55e9089efc15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7fdf1eac30ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fdf1eab7bdb testing::Test::Run() at ??:0
@ 0x7fdf1eab7d9d testing::TestInfo::Run() at ??:0
@ 0x7fdf1eab8377 testing::TestSuite::Run() at ??:0
@ 0x7fdf1eab877c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fdf1eac360d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fdf1eab7e63 testing::UnitTest::Run() at ??:0
@ 0x55e908a3fc7e RUN_ALL_TESTS() at ??:0
@ 0x55e908a3ee6b main at ??:0
@ 0x7fdf1c828bf7 __libc_start_main at ??:0
@ 0x55e9089d706a _start at ??:0
2025-10-25 20:10:59.646 UTC [31607] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 20:10:59.647 UTC [31711] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 20:10:59.648 UTC [31608] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 20:10:59.648 UTC [31603] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 20:10:59.651 UTC [31604] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 20:10:59.651 UTC [31605] FATAL: terminating connection due to unexpected postmaster exit